Method and apparatus for obtaining high dynamic range images

ABSTRACT

The application provides techniques for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor exposed to incident light from the scene for capturing an image. The image sensor has a multiplicity of light-sensing elements in an array and each light sensing element has a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and has a response function. Each light sensing element is responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array. Each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light sensing elements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.09/511,469, filed Feb. 23, 2000, now U.S. Pat. No. 7,084,905 which isincorporated by reference herein in its entirety and from which priorityis claimed.

This invention was made with government support under grant numberN00014-97-1-0553 awarded by the Office of Naval Research. The governmenthas certain rights in the invention.

BACKGROUND OF INVENTION

The present invention relates to method and apparatus for capturing animage of a scene, and, more particularly, to method and apparatus forcapturing a relatively high dynamic range image using a relatively lowdynamic range image sensor.

Virtually any real world scene produces a very large range of brightnessvalues. By contrast, known image sensing devices have very limiteddynamic ranges. For example, it is typical for a video image sensor toprovide only 8-bits or less of grey-scale or color information. In thecase of grey-scale images. 8-bits allows only 256 discrete grey-scalelevels which is not sufficient to capture the fine details of most reallife scenes.

A known solution to the problem of capturing high dynamic range imageswith a low dynamic range image sensor is to capture multiple images ofeach local area of the scene while varying the exposure to light fromthe scene. There are three general variations of this solution: (1)temporal variation of the exposure while capturing a sequence of images,(2) using multiple image sensors having different sensitivities to lightfrom the scene and (3) spatial variation of the exposure.

Typically, temporal exposure variation involves sequentially capturingmultiple images of the scene with different exposures and then combiningthe multiple captured images into a single high dynamic range image. Theuse of temporal exposure variation is disclosed, for example, in:Japanese Patent No. 08-331461 entitled “Driving Method for Solid-StateImage Pickup Device” to Hamazaki, issued December 1996; Japanese PatentNo. 07-254965 entitled “Electronic Image Pickup Device” to Saito, issuedFebruary 1995; U.S. Pat. No. 5,420,635 entitled “Video Camera, ImagingMethod Using Video Camera, Method of Operating Video Camera, ImageProcessing Apparatus and Method, and Solid-State Electronic ImagingDevice” to Konishi et al., issued May 1995; U.S. Pat. No. 5,455,621entitled “Image Method for a Wide Dynamic Range and an Imaging Devicefor a Wide Dynamic Range” to Morimura, issued October 1993; JapanesePatent No. 06-141229 entitled “Image Picking Up and Synthesizing Methodand Image Pickup Device with High Dynamic Range” to Morimura, issued May1994; U.S. Pat. No. 5,801,773 entitled “Image Data Processing Apparatusfor Processing Combined Image Signals in Order to Extend Dynamic Range”to Ikeda, issued September 1998; U.S. Pat. No. 5,638,118 entitled “ImageDevice with Diverse Storage Times Used in Image Composition” toTakahashi et al. issued June 1997; U.S. Pat. No. 5,309,243 entitled“Method and Apparatus for Extending the Dynamic Range of an ElectronicImaging System” to Tsai issued May 1994; Mann and Picard, “Being‘Undigital’ with Digital Cameras: Extending Dynamic Range by CombiningDifferently Exposed Pictures,” Proceedings of IST's 48th AnnualConference, pp. 422-428, May 1995; Debevec and Malik, “Recording HighDynamic Range Radiance Maps for Photographs,” Proceedings of the ACMSIGGRAPH, 1997, pp. 369-378, August 1997; and Mitsunaga and Nayar,“Radiometric Self Calibration,” IEEE Conference on Computer Vision andPattern Recognition (CVPR 99), pp. 374-380, June 1999. However, knowntemporal exposure variation techniques all have the fundamental problemin that changes in the scene may take place during the sequential imagecapture at different exposures. Consequently, these techniques areuseful only for static scenes where the radiance from the scene remainconstant and where the position and orientation of the image sensor, andits geometric characteristics remain unchanged during sequential imagecapture at different exposures.

Another known technique for capturing multiple images of a scene forhigh dynamic range imaging uses multiple optically aligned image sensorsand a beam splitter for providing incident light from the same view ofthe scene to the multiple image sensors. The image sensors havingdifferent sensitivities so that multiple images are simultaneouslycaptured at different exposures. The multiple images captured by theimage sensors are then combined to form a single high dynamic rangeimage. Such techniques, for example, are described in: Japanese PatentNo. 08-22341 entitled “Image Sensor” to Doi et al., issued August 1986;Japanese Patent No. 07-25495 entitled “Electronic Image Pickup Device”to Saito, issued February 1995; Japanese Patent No. 07-254966 entitled“Electronic Image Pickup Device” to Saito, issued October 1995; JapanesePatent No. 08-340486 entitled “Electronic Image Pickup Device to Saito,issued December 1996; Japanese Patent No. 10-069011 entitled “ImagePickup Device” to Kimura, issued March 1998; and U.S. Pat. No. 5,801,773entitled “Image Data Processing Apparatus for Processing Combined ImageSignals in Order to Extend Dynamic Range” To Ikeda, issued September1998. The multiple image sensor techniques have the drawback in thatthey require high precision optics and a plurality of image sensors sothat it is difficult to build compact and inexpensive imaging systemsusing these techniques.

Known spatial exposure variation techniques for high dynamic rangeimaging typically use a special image sensor having more than one(typically two) light-sensing elements having different sensitivitiesfor each pixel. The light-sensing elements for each pixel aresimultaneously exposed and their respective output signals are combinedto derive a brightness value for the pixel. Such techniques aredescribed, for example, in: U.S. Pat. No. 5,789,737 entitled “HighDynamic Range Segmented Pixel Sensor Array” to Street et al., issuedAugust 1998; U.S. Pat. No. 4,623,928 entitled “High Dynamic Range CCDDetector/Imager” to Handy, issued November 1986; U.S. Pat. No. 4,873,561entitled “High Dynamic Range Charge Coupled Device” to Wen, issuedOctober 1989; Japanese Patent No. 08-331461 entitled “Driving Method forSolid-State Image Pickup Device” to Hamazaki, issued December 1996;Japanese Patent No. 59-217358 entitled “Charge Coupling Image PickupDevice” to Murakoshi, issued December 1994; and U.S. Pat. No. 5,420,635entitled “Video Camera Imaging Method Using Video Camera, Method ofOperating Video Camera, Image Processing Apparatus and Method, andSolid-State Electronic Imaging Device” to Konishi et al., issued May1995. However, known spatial exposure variation techniques for highdynamic range imaging have the drawback of providing lower spatialresolution (typically a factor of two lower in the vertical direction)than the resolution of ordinary image sensors having the same number oflight-sensing elements.

Accordingly, there exists a need for a method and apparatus forcapturing a relatively high dynamic range images using a relatively lowdynamic range image sensor, which overcomes the problems of the priorart as discussed above.

SUMMARY OF THE INVENTION

In accordance with one aspect of the present invention, there isprovided a method for capturing a relatively high dynamic range image ofa scene using a relatively low dynamic range image sensor exposed toincident light from the scene. The image sensor has a multiplicity oflight-sensing elements in a linear or two-dimensional array, and eachlight-sensing element has a respective one of a plurality of sensitivitylevels to incident light in accordance with a predetermined sensitivitypattern for the array of light-sensing elements. Each of thelight-sensing elements in the array in response to incident light fromthe scene produces a captured image brightness value at a correspondingone of a multiplicity of pixel positions of a pixel position array,which may be a linear array having regularly spaced pixel positions in arow or a rectangular array having pixel positions at respectiveintersections of regularly spaced orthogonal pixel rows and pixelcolumns. In this manner each pixel position corresponds to a respectiveone of the plurality of sensitivity levels. The method comprises thestep of estimating respective off-grid brightness values at amultiplicity of off-grid positions from the captured image brightnessvalues at the multiplicity of pixel positions of the pixel positionarray, the off-grid positions being located in a regular array atrespective interstices of the pixel position array. The method furthercomprises the step of deriving pixel brightness values of a relativelyhigh dynamic range image of the scene from the estimated off-gridbrightness values at the multiplicity of off-grid positions. Where thepixel position array is a linear array in which the brightness value ateach pixel position is captured by a corresponding light-sensing elementof a linear image sensor, the off-grid positions are in a regular lineararray, and the relatively high dynamic range image is a linear imagerepresented by a regular linear array of pixel brightness values. Wherethe pixel position array is defined by respective intersections ofregularly spaced orthogonal pixel rows and pixel columns, the brightnessvalues of the pixel positions in the array may be captured bycorresponding light-sensing elements of a two-dimensional image sensoror may be a concatenation of the brightness values at pixel positions oflinear pixel position arrays representing line images of a scanned scenesuccessively captured by a linear image sensor.

In accordance with an exemplary embodiment of the method of the presentinvention, the off-grid positions are in a regular linear or rectangulararray and the step of deriving pixel brightness values of a relativelyhigh dynamic range image comprises using the estimated off-gridbrightness values as the pixel brightness values of the relatively highdynamic range image.

According to another exemplary embodiment of the method of the presentinvention, the step of deriving pixel brightness values of therelatively high dynamic range image comprises resampling the estimatedoff-grid brightness values at the multiplicity of off-grid positions toderive respective resampled brightness values at the multiplicity ofpixel positions of the pixel position array to thereby obtain the pixelbrightness values of the relatively high dynamic range image.

In accordance with still another exemplary embodiment of the method ofpresent invention, the step of estimating respective off-grid brightnessvalues at the multiplicity of off-grid positions from the captured imagebrightness values at the multiplicity of pixel positions comprises thesteps of estimating a sensitivity level off-grid brightness value foreach individual one of the plurality of sensitivity levels at each oneof the off-grid positions using only the captured image brightnessvalues at pixel positions corresponding to the individual one of thesensitivity levels. The sensitivity level off-grid brightness value foreach individual one of the sensitivity levels at each one of theoff-grid positions is multiplied by a predetermined weighting factor forthe individual one of the sensitivity levels to derive respectiveweighted sensitivity level off-grid brightness values for the differentones of the sensitivity levels at each one of the off-grid positions.The respective weighted sensitivity level off-grid brightness values forthe different ones of the sensitivity levels at each one of the off-gridpositions are added together to derive a weighted sum sensitivity levelbrightness value at each one of the off-grid positions. The weighted sumsensitivity level brightness value at each one of the off-grid positionsis compensated by the inverse of a combined response function of thelight-sensing elements to derive the respective estimated off-gridbrightness values at the off-grid positions. The combined responsefunction of the light-sensing elements is the sum of weighted responsefunctions of light-sensing elements having different ones of thesensitivity levels, each weighted response function being a responsefunction of a light-sensing element having a respective one of thesensitivity levels multiplied by a predetermined weighting factor forthe respective one of the sensitivity levels.

According to yet another exemplary embodiment of the method of thepresent invention, the step of estimating respective off-grid brightnessvalues at the multiplicity of off-grid positions from the captured imagebrightness values at the multiplicity of pixel positions comprisesestimating an off-grid brightness value at each individual one of theoff-grid positions by first compensating the captured image brightnessvalue at each individual one of a predetermined group of pixel positionsthat are nearest neighbors to the individual one of the off-gridpositions by the inverse of the response function of a light-sensingelement having a sensitivity level corresponding to the individual oneof the nearest neighbor pixel positions. The compensated captured imagebrightness values at the nearest neighbor pixel position are then eachcompared with a predetermined low threshold value and a predeterminedhigh threshold value, and a sum is taken of the compensated nearestneighbor pixel position brightness values that are greater than thepredetermined low threshold value and less than the predetermined highthreshold value. The sum is then divided by the number of compensatednearest neighbor pixel position brightness values included in the sum toderive an estimated off-grid brightness value for the individual one ofthe off-grid positions.

According to a further exemplary embodiment of the method of the presentinvention, interpolated on-grid brightness values at respective pixelpositions of the pixel position array are derived directly from thecaptured image brightness values using an on-grid interpolation filter.The interpolated on-grid brightness value at each individual one of thepixel positions is derived by computing the product of the capturedimage brightness values at a predetermined group of pixel positions thatare nearest neighbors to the individual one of the pixel positions andan on-grid interpolation filter kernel, the predetermined group of pixelpositions having the same dimensions as the filter kernel. Theinterpolated on-grid brightness values are then each compensated by theinverse of a combined response function of light-sensing elements havingdifferent sensitivity levels to derive the pixel brightness values ofthe relatively high dynamic range output image.

In accordance with another aspect of the present invention, there isprovided a system for capturing a relatively high dynamic range image ofa scene using a relatively low dynamic range image sensor adapted to beexposed to incident light from the scene. The system comprises an imagesensor having a multiplicity of light-sensing elements in a linear ortwo-dimensional array. Each light-sensing element in the array having arespective one of a plurality of sensitivity levels to incident light inaccordance with a predetermined sensitivity pattern for the array oflight-sensing elements. Each light-sensing element being responsive toincident light from the scene for producing a captured image brightnessvalue at a corresponding one of a multiplicity of pixel positions of apixel position array, which may be a linear array having regularlyspaced pixel positions in a row or a rectangular array having pixelpositions located at respective intersections of regularly spacedorthogonal pixel rows and pixel columns. In this manner, each pixelposition corresponds to a respective one of the plurality of sensitivitylevels of the light-sensing elements. The system further comprises acaptured image memory for storing the respective captured imagebrightness values at the multiplicity of pixel positions produced by thelight-sensing elements, and an off-grid estimator for deriving from thecaptured image brightness values in the captured image memory respectiveestimated off-grid brightness values at a multiplicity of off-gridpositions located in a regular array at respective interstices of thepixel position array. The system further includes an output imagegenerator for deriving pixel brightness values of a relativelyhigh-dynamic range output image from the estimated off-grid brightnessvalues, and an output image memory for storing the pixel brightnessvalues of the output image. Where the pixel position array is a lineararray and the captured image memory stores brightness values captured bycorresponding light-sensing elements of a linear image sensor, theoff-grid positions are in a regular linear array and the output image isa linear image represented by a regular array of pixel brightnessvalues. Where the pixel position array is a rectangular array defined byrespective intersections of regularly spaced orthogonal pixel rows andpixel columns, the captured image memory may store brightness valuescaptured by corresponding light-sensing elements of a two-dimensionalimage sensor or a concatenation of linear pixel position arraybrightness values of line images of a scanned scene successivelycaptured by a linear image sensor. In the latter instance, the off-gridpositions are in a regular two-dimensional array and the output image isa two-dimensional image represented by a regular two-dimensional arrayof pixel brightness values.

According to an exemplary embodiment of the system of the presentinvention, the output image generator provides the estimated off-gridbrightness values derived by the off-grid estimator to the output imagememory as the pixel brightness values of the output image.

According to another exemplary embodiment of the system of the presentinvention, the system includes an off-grid brightness value memory forstoring the respective estimated off-grid brightness values derived bythe off-grid estimator at the off-grid positions, and an on-gridresampler for deriving from the estimated off-grid brightness values inthe off-grid brightness value memory respective resampled brightnessvalues at the pixel positions of the pixel position array. The resampledbrightness values are then provided to the output image memory forstorage therein as the pixel brightness values of the output image.

According to still another exemplary embodiment of the system of thepresent invention, the off-grid estimator include an off-grid positiongenerator for providing the multiplicity of off-grid positions and asensitivity pattern memory for storing the respective sensitivity levelscorresponding to the pixel positions. The off-grid estimator alsoincludes a plurality of sensitivity level off-grid brightness valueestimators, each corresponding to a respective one of the plurality ofsensitivity levels of the light-sensing elements, for receiving off-gridpositions from the off-grid position generator and sensitivity leveldata from the sensitivity pattern memory, and for deriving from thecaptured image brightness values in the captured image memory aplurality of sensitivity level off-grid brightness values eachcorresponding to a different one of the plurality of sensitivity levelsat each one of the off-grid positions received from the off-gridposition generator. The off-grid estimator also includes an accumulatorfor receiving the sensitivity level off-grid brightness values derivedby the plurality of sensitivity level off-grid brightness valueestimators at each one of the off-grid positions. The accumulatorcomprises a weighting factor memory for storing respective ones of aplurality of predetermined weighting factors each corresponding to arespective one of the plurality of sensitivity levels, and a pluralityof multipliers each for computing the product of the sensitivity leveloff-grid brightness value corresponding to a respective one of theplurality of sensitivity levels and the weighting factor correspondingto the same sensitivity level to provide a respective weightedsensitivity level off-grid brightness value corresponding to therespective one of the sensitivity levels at each one of the off-gridpositions received from the off-grid position generator. The accumulatorfurther includes an adder for summing the weighted sensitivity leveloff-grid brightness values corresponding to respective ones of thesensitivity levels at each one of the received off-grid positionscomputed by the plurality of multipliers to derive a respective weightedsum sensitivity level off-grid brightness value at each one of thereceived off-grid positions. The off-grid estimator further comprises aresponse function compensator for compensating each one of the weightedsum sensitivity level off-grid brightness values by the inverse of acombined response function of light-sensing elements to derive therespective estimated off-grid brightness values at the off-gridpositions. The combined response function of the light-sensing elementsis the sum of weighted response functions of light-sensing elementshaving different ones of the sensitivity levels, each one of theweighted response functions being a response function of a light-sensingelement having a respective one of the plurality of sensitivity levelsmultiplied by a predetermined weighting factor corresponding to therespective one of the plurality of sensitivity levels.

In yet another exemplary embodiment of the system of the presentinvention, the off-grid estimator comprises a pixel position generatorfor providing the multiplicity of pixel positions, a sensitivity patternmemory for storing respective sensitivity levels corresponding to themultiplicity of pixel positions, and a response function compensatorcoupled to receive pixel positions from the pixel position generator andcorresponding sensitivity levels from the sensitivity pattern memory.The response function compensator also receives captured imagebrightness values from the captured image memory at the pixel positionsreceived from the pixel position generator and compensates the receivedcaptured image brightness values by the inverse of the responsefunctions of light-sensing elements having sensitivity levelscorresponding to the received pixel positions to provide respectivecompensated captured image brightness values at the pixel positionsreceived from the pixel position generator. The off-grid estimator alsoincludes a compensated on-grid brightness value memory for storingrespective compensated captured image brightness values at respectivepixel positions, an off-grid position generator for providing themultiplicity of off-grid positions and a pseudoinverse estimator. Thepseudoinverse estimator receives off-grid positions from the off-gridposition generator and derives from the compensated captured imagebrightness values in the compensated on-grid brightness value memory therespective estimated off-grid brightness values at the received off-gridpositions.

In a further exemplary embodiment of the system of the present inventionwhere the system includes an off-grid brightness value memory forstoring the respective estimated off-grid brightness values derived bythe off-grid estimator at the multiplicity of off-grid positions and anon-grid resample for deriving from the estimated off-grid brightnessvalues in the off-grid brightness value memory respective resampledbrightness values at the multiplicity of pixel positions of the pixelposition array, the on-grid resampler comprises a pixel positiongenerator for providing the multiplicity of pixel positions, aninterpolator for receiving pixel positions from the pixel positiongenerator and deriving a respective resampled brightness value for eachone of the received pixel positions by computing the product of theestimated off-grid brightness values at a predetermined number ofoff-grid positions that are nearest neighbors to the received pixelpositions and an interpolation kernel having the same dimensions as thepredetermined number of nearest neighbor off-grid positions.

In a still further exemplary embodiment of the system of the presentinvention, there is provided a system for capturing a relatively highdynamic range image of a scene using a relatively low dynamic rangeimage sensor adapted to be exposed to incident light from the scene. Thesystem comprising an image sensor having a multiplicity of light-sensingelements in a linear or two-dimensional array, each light-sensingelement having a respective one of a plurality of sensitivity levels inaccordance with a predetermined sensitivity pattern for thelight-sensing element array, and being responsive to incident light fromthe scene for producing a captured image brightness value at acorresponding one of a multiplicity of pixel positions, where the pixelposition array is rectangular, the captured image brightness values maybe produced by corresponding light-sensing elements of a two-dimensionalimage sensor, or may be a concatenation of linear brightness valuearrays of line images of a scanned scene successively captured by imagesensor. The system further comprises a captured image memory for storingthe captured image brightness values produced by the light-sensingelements at respective pixel positions of a pixel position array, whichmay be a linear array having regularly spaced pixel positions in a rowor a rectangular array having pixel positions located at respectiveintersections of regularly spaced orthogonal pixel rows and pixelcolumns. The system also comprises a pixel position generator forproviding the pixel positions of the pixel position array and an on-gridinterpolation filter receiving pixel positions from the pixel positiongenerator for deriving from the captured image brightness values in thecaptured image memory interpolated on-grid brightness values at thepixel positions received from the pixel position generator. The filtercomputes the product of respective captured image brightness values at apredetermined number of nearest neighbor pixel positions to each pixelposition received from the pixel position generator and an on-gridinterpolation filter kernel to derive an interpolated on-grid brightnessvalue at each pixel position received from the pixel position generator,the dimensions of the on-grid interpolation filter being the same as thedimensions of the predetermined number of nearest neighbor pixelpositions. Also included in the system is a response functioncompensator for compensating the interpolated on-grid brightness valueat each one of the received pixel positions by the inverse of a combinedresponse function of the light-sensing elements to derive a compensatedinterpolated on-grid brightness value at the received pixel position,and an output image memory for storing the compensated interpolatedon-grid brightness values at the respective pixel positions of the pixelposition array as the pixel brightness values of a high dynamic rangeoutput image. The combined response function of the light-sensingelements is the sum of weighted response functions of light-sensingelements having different ones of the plurality of sensitivity levels,each one of the weighted response functions being the response functionof a light-sensing element having a respective one of the plurality ofsensitivity levels multiplied by a predetermined weighting factor forthe respective one of the plurality of sensitivity levels.

BRIEF DESCRIPTION OF DRAWINGS

For a more complete understanding of the nature and benefits of thepresent invention, reference should be made to the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a data flow diagram representing a system for obtaining highdynamic range images according to the present invention;

FIG. 2 is a data flow diagram representing the image processing part ofthe data flow diagram of FIG. 1 according to an exemplary embodiment ofthe present invention;

FIG. 3 is a block diagram depicting the hardware components of anexemplary system in accordance with the present invention;

FIG. 4 depicts an illustrative mask having a spatial repetition ofgroups of four cells where each cell of each group has a differenttransparency in a common predetermined positional order;

FIG. 5 is a group of four cells having different transparencies in apredetermined positional order of the mask of FIG. 4;

FIG. 6 illustrates the capture of a scene having a bright object and adark object using a masked image sensor in accordance with the presentinvention;

FIG. 7 is an illustrational representation of a captured image of ascene by a masked image sensor in accordance with the present invention;

FIG. 8 is a schematic representation of the cross section of alight-sensing element and associated structure of a typical chargecouple device image sensor;

FIG. 9 is a schematic representation of the cross section of twoadjacent light-sensing elements and associated structure of a chargecoupled device image sensor illustrating one technique for providing thelight-sensing elements with different sensitivity levels;

FIG. 10 is a schematic representation of the cross section of twoadjacent light-sensing elements and associated structure of a chargecoupled device image sensor illustrating another technique for providingthe light-sensing elements with different sensitivity levels;

FIG. 11 is a schematic representation of the cross section of twoadjacent light-sensing elements and associated structure of a chargecoupled device image sensor illustrating yet another technique forproviding the light-sensing elements with different sensitivity levels;

FIG. 12 is a schematic representation of a simplified charge coupleddevice image sensor having an interline structure showing terminals forapplying reset trigger pulses and transfer trigger pulses for respectivelight-sensing elements to provide the light-sensing elements withdifferent sensitivity levels;

FIG. 13 is a schematic representation of the cross section of alight-sensing element, its associated structure and a stage of acorresponding vertical shift register of a typical charge coupled deviceimage sensor having an interlined structure illustrating the sinking ofstored photogenerated charge upon the application of a reset triggersignal and the transfer of stored photogenerated charge to a respectiveshift register stage upon the application of a transfer trigger signal;

FIG. 14 show timing diagrams depicting waveforms of an exposure pulse,reset trigger pulses, transfer trigger pulses for four photodiodes andthe magnitude of charge stored in each of the four photodiodes for thesimplified charge coupled device image sensor depicted in FIG. 13 toprovide different sensitivity levels for the four light-sensing elementsof the image sensor;

FIG. 15 is a diagram illustratively depicting pixel positions located atrespective intersections of a plurality of regularly spaced pixel rowsand a plurality of regularly spaced pixel columns orthogonal to thepixel rows, and off-grid positions located at respective intersectionsof a plurality of regularly spaced intermediate rows and a plurality ofregularly spaced intermediate columns;

FIG. 16 is a data flow diagram of the off-grid estimator of the diagramof FIG. 2 according to another exemplary embodiment of the presentinvention;

FIG. 17 is a flow diagram describing a computation process that may becarried out by the embodiment of the off-grid estimator of FIG. 16;

FIG. 18 is a flow diagram describing the computation process forevaluating the Resample_(i) (I_(c),S,x,y) function used in thecomputation process described by the flow diagram of FIG. 17;

FIG. 19 is a flow diagram of the computation process for evaluating theMap (LUT,s) function used in the computation process described by theflow diagram of FIG. 17;

FIG. 20 graphically illustrates examples of radiometric responsefunctions of four light-sensing elements having four differentsensitivity levels;

FIG. 21 graphically illustrates an exemplary combined radiometricresponse function of the light-sensing elements derived by computing theweighted sum of the exemplary radiometric response functions oflight-sensing elements having four different sensitivity levels depictedin FIG. 20;

FIG. 22 graphically illustrates lookup table data derived from theinverse of the combined radiometric response function of FIG. 21;

FIG. 23 graphically illustrates lookup table data for an inverse of acombined radiometric response function in which gamma characteristicsare included;

FIG. 24 is a data flow diagram of the off-grid estimator of the diagramof FIG. 2 according to yet another exemplary embodiment of the presentinvention;

FIG. 25 is a flow diagram describing a computation process that may becarried out by the embodiment of the off-grid estimator represented bythe data flow diagram of FIG. 24;

FIG. 26 is a flow diagram describing an alternative computation processthat may be carried out by the embodiment of the off-grid estimatorrepresented by the data flow diagram of FIG. 24;

FIG. 27 shows an example of a 4×4 cubic interpolation filter kernel;

FIG. 28 is a data flow diagram of the on-grid resampler of the data flowdiagram of FIG. 2 according to still another exemplary embodiment of thepresent invention;

FIG. 29 is a diagram of pixel positions and off-grid positions forexplaining on-grid resampling using a 4×4 cubic interpolation filter;

FIG. 30 is a flow diagram describing a computation process that may becarried out by the embodiment of the on-grid resampler represented bythe data flow diagram of FIG. 28;

FIG. 31 shows an example of a bi-cubic B-spline interpolation filterkernel;

FIG. 32 shows an example of a bi-linear interpolation filter kernel;

FIG. 33 shows an example of a 6×6, 2-dimensional Gaussian interpolationfilter kernel;

FIG. 34 shows an example of a simple shift filter kernel;

FIG. 35 depicts an illustrative mask having cells with two differenttransparencies for providing an image sensor with a spatially varyingsensitivity pattern with two sensitivity levels;

FIG. 36 is a diagram illustratively depicting pixel positions located atrespective intersections of a plurality of regularly spaced pixel rowsand a plurality of regularly spaced pixel columns orthogonal to thepixel rows, and off-grid positions located at respective intersectionsof the pixel rows and a plurality of regularly spaced intermediatecolumns and at respective intersections of the pixel columns and aplurality of regularly spaced intermediate rows;

FIG. 37 is a data flow diagram of the off-grid estimator of the dataflow diagram of FIG. 2 for an image sensor having two sensitivity levelsin accordance with a further exemplary embodiment of the presentinvention;

FIG. 38 is a flow diagram describing a computation process that may becarried out by the embodiment of the off-grid estimator for twosensitivity levels represented by the flow diagram of FIG. 37;

FIG. 39 is a flow diagram describing a computation process that may Decarried out by the exemplary embodiment represented by the data flowdiagram of FIG. 24 to provide an exemplary embodiment of a twosensitivity level off-grid estimator with a cubic pseudoinverseestimator according to the present invention;

FIG. 40 is a flow diagram describing a computation process forevaluating the function Pseudoinverse (I_(m),x′,y′) used in thecomputation process described in the flow diagram of FIG. 39;

FIG. 41 is a diagram showing a shaded region of pixel positions andoff-grid positions for explaining on-grid resampling using a 45° rotatedcubic interpolation filter kernel for an exemplary embodiment of theon-grid resampler for two sensitivity levels in accordance with thepresent invention;

FIG. 42 shows an example of a 45° rotated cubic interpolation filterkernel that may be used in the computation process described by the flowdiagram of FIG. 40 for an exemplary embodiment of the on-grid resamplerfor two sensitivity levels;

FIG. 43 is a flow diagram describing the computation process that may becarried out by the exemplary embodiment of the on-grid resamplerrepresented by the data flow diagram of FIG. 28 to obtain an exemplaryembodiment of the on-grid resampler for two sensitivity levels inaccordance with the present invention;

FIG. 44 is a data flow diagram of still further exemplary embodiment ofthe image processing part of the data flow diagram of FIG. 1;

FIG. 45 is a detailed data flow diagram of yet further exemplaryembodiment of the image processing part of the data flow diagram of FIG.1 for four sensitivity levels;

FIG. 46 is a flow diagram describing the computation process that may becarried out by the exemplary embodiment of the image processing partrepresented by the data flow diagram of FIG. 45;

FIG. 47 is a data flow diagram of still another exemplary embodiment ofthe image processing part of the data flow diagram of FIG. 1;

FIG. 48 is a flow diagram describing the computation process that may becarried out by the exemplary embodiment of the image processing partrepresented by the flow diagram of FIG. 47;

FIG. 49 shows an example of a 5×5 filter kernel that may be used in thecomputation process described by the flow diagram of FIG. 48;

FIG. 50 illustrates the capture of successive line images of a scene byan imaging system having a linear image sensor, the imaging systemscanning the scene by moving at constant speed with respect to thescene, the linear image sensor taking sequential exposures of the sceneat regular time intervals;

FIG. 51 illustrates the capture of successive line images of objects ona conveyor belt moving at constant speed past a stationary imagingsystem having a linear image sensor taking sequential exposures of theconveyor belt and objects at regular time intervals;

FIG. 52 depicts an illustrative exposure mask having cells with twodifferent transparencies for providing a linear image sensor with aspatially varying sensitivity pattern with two sensitivity levels;

FIG. 53 depicts illustratively a concatenation of the exposure mask ofFIG. 52 representing a two-dimensional spatially varying sensitivitypattern of a two-dimensional image formed by concatenating successiveline images captured by a linear image sensor having a one-dimensionalspatially varying sensitivity pattern defined by the exposure mask ofFIG. 52;

FIG. 54 is a diagram illustratively depicting a linear array of pixel(on-grid) positions and a linear array of off-grid positions;

FIG. 55 is a flow diagram describing a computation process that may becarried out by the embodiment of the off-grid estimator represented bythe flow diagram of FIG. 24 for a line image captured by a linear imagesensor having light-sensing elements of two sensitivity levels;

FIG. 56 shows an example of a 4×1, one-dimensional Gaussianinterpolation kernel that may be used in the computation processdescribed by the flow diagram of FIG. 55;

FIG. 57 is a flow diagram describing a computation process that may becarried out by the embodiment of the on-grid resampler represented bythe data flow diagram of FIG. 28 to derive on-grid pixel brightnessvalues of a one-dimensional output image;

FIG. 58 shows an example of a 4×1 one-dimensional cubic interpolationkernel that may be used in the computation process described by the flowdiagram of FIG. 57;

FIG. 59 is a data flow diagram representing still another exemplaryembodiment of the image processing part of the data flow diagram of FIG.1;

FIG. 60 is a flow diagram describing a computation process that may becarried out by the embodiment of the image processing part representedby the data flow diagram of FIG. 59 to derive on-grid pixel brightnessvalues of a linear output image; and

FIG. 61 shows an example of a 5×1 on-grid interpolation filter kernelthat may be used in the computation process described by the flowdiagram of FIG. 60.

Throughout the figures of the drawings the same reference numerals andcharacters are used to designate common features components, parts,processes and steps.

DETAILED DESCRIPTION

Referring to FIG. 1, there is shown a data flow diagram 100 of a systemfor capturing relatively high dynamic range images using a relativelylow dynamic range image sensor in accordance with the present invention.The system has an image sensing part 1 that includes a relatively lowdynamic range image sensor (not shown in FIG. 1) having a multiplicityof light-sensing elements in an array (not shown). The term “light” asused in the specification and claims shall mean electromagneticradiation whose wavelength may be anywhere in the spectrum from the farinfrared to x-rays. The array of light-sensing elements of the imagesensor may have any arrangement, including without limitationrectangular, linear, circular and hexagonal grids, and the light-sensingelements in the array need not be uniformly disposed. The array oflight-sensing elements has a predetermined spatially varying sensitivitypattern to incident light, and each light-sensing element of the imagesensor produces in response to incident light from a scene a capturedimage brightness value at a corresponding one of a multiplicity of pixelpositions of a pixel position array defined by respective intersectionsof a plurality of regularly spaced pixel rows and a plurality ofregularly spaced pixel columns orthogonal to the pixel rows (not shownin FIG. 1). The captured image brightness values at respective pixelpositions produced by the light-sensing elements are stored in acaptured image memory (not shown in FIG. 1) in the image processing part2 of the system. The image processing part 2 serves to remove thevariation in image brightness caused by the spatially varyingsensitivity pattern of the image sensor in the image sensing part 1 andprovides resampled brightness values at respective pixel positions,representing a relatively high dynamic range output image, to the outputimage memory 3.

Turning to FIG. 2, there is shown a dataflow diagram 200 of the imageprocessing part 2 of the data flow diagram of FIG. 1. The captured imagebrightness values at respective pixel positions of the pixel positionarray, as provided by the image sensor, are stored in a captured imagememory 20. The captured image brightness values in the captured imagememory 20 are used by an off-grid estimator 22 to derive estimatedoff-grid brightness values at off-grid positions located at respectiveinterstices of the pixel position array. The estimated off-gridbrightness values at respective off-grid positions are then resampled byan on-grid resampler 23 to derive resampled brightness values atrespective pixel positions. The resampled brightness values are storedin an output image memory 21 as the pixel brightness value representingthe relatively high dynamic range image. Because the brightness valueproduced by each one of the light-sensing elements of the image sensoris not combined with the brightness value produced by any otherlight-sensing element, there is no reduction in the resolution of theimage captured by the image sensor. Also, because the brightness valuesat respective off-grid positions are estimated from the captured imagebrightness values at respective pixel positions and the resampledbrightness values at respective pixel positions are derived byresampling of the estimated off-grid brightness values at the off-gridpositions, the captured image brightness value produced by any one ofthe light-sensing elements can contribute to any one of the resampledbrightness values. Consequently, the resampled output image in theoutput image memory 21 has almost the same resolution as the capturedimage in the captured image memory 20.

Referring to FIG. 3, there is shown a block diagram of the hardwarecomponents of a system 300 according to an exemplary embodiment of thepresent invention. The system 300 has an image sensing part 1, an imageprocessing part 2 and video output part 4. A scene is captured by theimage sensing part 1 with an image sensor having a predeterminedspatially varying sensitivity pattern. The image sensing part 1 providesan image signal representing the captured image to the image processingpart 2. The image processing part removes the variations in brightnessof the image caused by the spatially varying sensitivity pattern of theimage sensor and provides an output image signal to the video outputpart 4. The video output part 4 is not a part of the present inventionbut is included here for the purpose of showing a complete video imagingsystem that provides an output video signal representing relatively highdynamic range images.

The image sensing part 1 includes a lens system 6, an aperture 7, acharge coupled device (CCD) image sensor 9 and a preamplifier 10.Incident light 5 from a scene passes through the lens system 6 and theaperture 7 before reaching the CCD image sensor 9. The CCD image sensor9 has a predetermined spatially varying sensitivity pattern according towhich the sensitivity level of each of the light-sensing elements (notshown) of the image sensor 9 is determined. Different exemplary ways ofproviding a spatially varying sensitivity pattern for the CCD imagesensor 9 are described hereinbelow. The preamplifier 10 consists of agroup of several analog circuits which are typically used in videocameras, such as an automatic gain control circuit for adjusting theamplitude of the CCD image sensor output and a correlated doublesampling circuit for reducing noise.

The image sensing part 2 has an analog-to-digital (A/D) converter 11, aframe memory 12, a digital-to-analog (D/A) converter 13, a read-onlymemory (ROM) 14, a random-access memory (RAM) 15 and a processor unit16. The A/D converter 11 digitizes the image signal from the imagesensing part 1, and the digitized image data (captured image brightnessvalues at respective pixel positions) are stored in the frame memory 12.Computations in the image processing part 2 are performed by theprocessing unit 16, which has access to the data in the frame memory 12,the ROM 14 and the RAM 15. Predefined data and programs to be used bythe processing unit 16 to perform its computations are stored in the ROM14. A portion of the RAM 15 is used for temporary storage of data duringcomputations by the processing unit 16. Computed resampled brightnessvalues at respective pixel positions, which are stored in the framememory 12, are converted to an analog image signal by the D/A converter13 and provided to the video output part 4.

The video output part 4 comprises a video encoder 17 and an output unit18. The analog image signal from the image processing part 2 is encodedto a video signal by the video encoder 17, and the output unit 18outputs the video output signal 19.

Image Sensing Part

The spatially varying sensitivity pattern of the CCD image sensor 9 maybe implemented by using a fixed pattern mask or by on-chip techniques.Turning now to FIG. 4, there is shown an example of a fixed pattern mask101 having a predetermined spatially varying exposure pattern. In thisexample, the fixed pattern mask 101 is a plate having a two-dimensionalarray of cells of different transparencies (attenuations). The exposurepattern of the mask 101 is defined by a repetitive pattern of identicalgroups of four nearest neighbor cells each having a differenttransparency. An example of such a group of four nearest neighbor cellsis shown in FIG. 5. The group has a most transparent cell 102, atransparent cell 103, a less transparent cell 104 and a leasttransparent cell 105 arranged in a predetermined positional order commonto all groups of cells of the mask 101. Each cell of the mask 101 isused to control the sensitivity level of a respective light-sensingelement of the image sensor. In the case of a CCD image sensor, the mask101 may be fabricated on the same substrate as the CCD device by forminga transparent layer over the photosensitive region of the device andforming the cells of different transparencies by patterning the layerusing known photolithography and etching techniques to form a mask witha desired exposure pattern. Alternatively, the light-sensing surfaces ofthe light-sensing elements themselves may be subjected to masked etchingoperations to alter the photosensitivities of preselected light-sensingelements to desired levels. By using the fixed pattern mask 101, eachlocal area of a captured image exposed through one of the groups of fourneighboring cells 102, 103, 104 and 105 will have four differentexposures. In this manner, every such local area of the captured imagewill have a relatively high dynamic range resulting from the use of fourdifferent exposures (i.e., four light-sensing elements having differentsensitivity levels).

Referring to FIG. 6, there is illustrated an arrangement for capturingthe image of a scene using the fixed pattern mask 101. As shown in thefigure, the fixed pattern mask 101 is placed directly in front of animage sensor 108. The mask 101 is aligned with the image sensor 108 sothat each cell of the mask 101 is positioned directly in front of acorresponding light-sensing element (not shown) of the image sensor 108.Incident light from the scene passes through the camera optics 109 andis then focused onto the image plane of the image sensor 108 through themask 101. The brightness of the incident light which is detected by eachlight-sensing element of the image sensor 108 is determined by thetransparency of the mask cell which is in front of that light-sensingelement. In the scene which is being captured in FIG. 6, there is abright object 106 and dark object 107.

Turning to FIG. 7, there is shown a representation 110 of the image ofthe scene in FIG. 6 as captured by the image sensor 108 through thefixed pattern mask 101. It is noted in FIG. 7 that even if pixels ofbrighter exposure have saturated brightness values in an area of thebright object 106, pixels of darker exposure in the same area havenon-saturated brightness values. Similarly, even if pixels of darkerexposure have brightness values below the noise level of thelight-sensing elements receiving incident light from the area of thedark object 107 in FIG. 6, pixels of brighter exposure receivingincident light from the same area will have brightness values above thenoise level.

Referring to FIG. 8, there is shown in schematic representation 800 ofthe cross section of a light-sensing element and associated structure ofa typical CCD image sensor, which is the preferred type of image sensorfor implementing a system according to the present invention. Thelight-sensing element comprises a photodiode consisting of an n-typeregion 115 formed within a p-well region 190. The photodiode has alight-sensing surface 191 adapted to be exposed to incident light and isoperatively biased (n-type region 115 being reverse biased with respectto the p-well region 190) to form a potential well in the p-well regionfor storing photogenerated charge therein. Associated with thephotodiode is a transfer gate electrode 114, which upon the applicationof an appropriate voltage signal thereto creates a channel for thephotogenerated charge stored in the photodiode to pass to a verticalshift register (not shown in FIG. 8), as will be further explainedhereinbelow. An opaque light shield layer 113 is formed to preventincident light from reaching regions other than the light-sensingsurfaces of the photodiodes of the CCD image sensor. An aperture 192 isformed in the light shield layer 113 to expose the light-sensing surface191 of each one of the photodiodes in the CCD image sensor. An on-chipmicro lens 111 may be formed above the light-sensing surface 191 of thephotodiode to concentrate incident light onto the light-sensing surface191. Especially for color CCD image sensors, there is formed an on-chipoptical filter 112 between the on-chip micro lens 111 and thelight-sensing surface 191 of the photodiode.

Turning to FIG. 9, there is shown a schematic representation 900 of thecross section of two adjacent light-sensing elements and associatedfeatures of a CCD image sensor, in which the two light-sensing elementshave different sensitivity levels. The light-sensing element on the leftof the figure has an on-chip micro lens 111 for concentrating incidentlight onto the light sensitive surface 191 of photodiode, while thelight-sensing element on the right does not have an on-chip micro lensabove the light-sensing surface 191 of its photodiode. Therefore, thelight-sensing element on the left will receive a greater intensity ofincident light at its light-sensing surface 191 than the light-sensingelement on the right. In this manner the light-sensing element on theleft is fabricated to have a higher sensitivity level than thelight-sensing element on the right. By forming an on-chip micro lens 111above preselected light-sensing elements of the image sensor apredetermined spatially varying sensitivity pattern having twosensitivity levels is attained for the CCD image sensor.

Turning to FIG. 10, there is a shown a schematic representation 1000 ofthe cross section of two adjacent light-sensing elements and associatedstructure of a CCD image sensor, illustrating an alternative techniquefor providing the light-sensing elements with different sensitivitylevels. Respective on-chip filters 116 and 117 are formed between theon-chip micro lenses 111 and the light-sensing surfaces 191 of the rightand left photodiodes, respectively. The filters 116 and 117 are formedto have different transparencies to incident light, thus causing thelight-sensing elements on the right and left to have differentsensitivity levels. By forming the filters above the photosensitivesurfaces 191 of respective photodiodes of the CCD image sensor to havepredetermined transparencies, a predetermined spatially varyingsensitivity pattern is attained for the CCD image sensor.

Referring to FIG. 11, there is shown a schematic representation 1100 ofthe cross section of two adjacent light-sensing elements and associatedstructure of a CCD image sensor. The light shields layer 113 hasrespective apertures 118 and 119 above the photosensitive surfaces 191of the left and right light-sensing elements. The apertures 118 and 119in the light shield layer 113 are formed to have different sizes so asto allow different amounts of incident light to reach the light-sensingsurfaces 191 of the left and right photodiodes. By forming the aperturesabove the light-sensing surfaces 191 of respective light-sensingelements of the image sensor to have predetermined sizes, apredetermined spatially varying sensitivity pattern is attained for theimage sensor.

Referring to FIG. 12, there is shown a schematic layout diagram 1200representing a simplified CCD image sensor having four photodiodes 121,122, 123 and 124, and an interline structure in which each column ofphotodiodes 121 and 123, and 124 and 122 is located adjacent acorresponding vertical shift register 125 and 126, respectively. Asexplained above in connection with FIG. 8, each photodiode has anassociated transfer gate electrode 114. When an appropriate voltagetransfer trigger signal is applied to the transfer gate electrode 114associated with a photodiode, photogenerated charge stored in thephotodiode is allowed to pass to a respective stage of the correspondingvertical shift register and is accumulated therein for the duration ofan exposure period. The diagram of FIG. 12 shows four transfer triggerinputs 129, 130, 131 and 132 for applying transfer trigger signals tothe transfer gate electrodes associated with photodiodes 121, 122, 123and 124, respectively, to transfer photogenerated charge stored in thephotodiodes 121, 122, 123 and 124 to respective stages of correspondingshift registers 125 and 126. At the end of each exposure period storedphotogenerated charge transferred from the photodiodes and accumulatedin respective stages of the vertical shift registers are shifted in eachvertical shift register 125 and 126 by a three-phase clock (not shown)to respective stages of a horizontal shift register 127. Once in thehorizontal shift register 127, the accumulated photogenerated chargefrom each one of the photodiodes 121, 122, 123 and 124 are shifted outsequentially by a two phase clock (not shown) to an output terminal 128where the respective charge packets are detected as voltage signalproportional to the magnitude of the accumulated photogenerated chargepacket reaching the output terminal 128.

Turning to FIG. 13, there is shown a schematic representation 1300 ofthe cross section of a photodiode formed by n-type region 115 and p-wellregion 190, a stage of the corresponding vertical shift register formedby n-type region 135 and p-well region 190, a p-type barrier region 134separating the photodiode and the shift register stage, and the transfergate electrode 114 associated with the photodiode connected to atransfer trigger input 137. The shift register stage comprises an n-typeregion 135 which is appropriately biased with respect to the p-well 190to form a potential well (not shown) for receiving and accumulatingphotogenerated charge transferred to the shift register stage. When atransfer trigger pulse signal is applied to terminal 137, photogeneratedcharge stored in the potential well of the photodiode form by n-typeregion 115 and the p-well 190 is allowed to pass through a channelbeneath the barrier region 134 to the potential well of the shiftregister stage, as indicated by the arrow 138.

In addition to the periodic application of a transfer trigger pulse totransfer trigger input 137, the photogenerated charge stored in eachphotodiode is periodically discharged by the periodic application of areset pulse signal to a reset input 133. The application of each resetpulse voltage signal appropriately biases the junction between thep-well and a heavily doped n-type charge sink region 120 to causesubstantially all of the photogenerated charge stored in the potentialwell of each photodiode of the CCD image sensor to be transferred to thecharge sink region 120, as indicated by the arrow 139.

Turning now to FIG. 14, there are shown timing diagrams of an exposurepulse 193, reset trigger pulses 194, transfer trigger pulses 195, 198,261 and 264 associated with the four photodiodes of the exemplary CCDimage sensor of FIG. 12, and the magnitudes of the stored photogeneratedcharge 196, 199, 262 and 265 in the four photodiodes PD4, PD3, PD2 andPD1, respectively, of the exemplary CCD image sensor. The duration ofthe exposure period for capturing an image is determined by the width ofthe exposure pulse 193. Each time a reset pulse 194 is applied to thereset trigger input 133 of the CCD image sensor, the photogeneratedcharge stored in each photodiode of the image sensor goes to zero forreasons explained hereinabove. Because all photodiodes in the CCD imagesensor are exposed to incident light the magnitude of the photogeneratedcharge, as represented by waveforms 196, 199, 262 and 265, increasesbetween reset pulses 194. As explained above, each time a transfertrigger pulse is applied to the transfer gate electrode associated witha photodiode, photogenerated charge stored in the photodiode istransferred to a respective stage of a corresponding vertical shiftregister. In the case of photodiode PD4, transfer trigger pulses 195 areapplied to its associated transfer gate electrode at the same frequencyas the reset pulses 194 but with the phase shifted so that each transfertrigger pulse 195 immediately precedes the next reset pulse 194. Theapplication of each transfer trigger pulse 195 causes an amount ofcharge represented by a corresponding blackened region 197 to betransferred to a respective stage of a corresponding vertical shiftregister, and the transferred charge is accumulated therein for theduration of the exposure pulse 193.

Transfer trigger pulses 198, which have the same width as transfertrigger pulses 195, are applied to the transfer gate electrodeassociated with photodiode PD3 at half the frequency as that of transfertrigger pulses 195. The phase of the transfer trigger pulses 198 is suchthat each transfer trigger pulse 198 immediately precedes the next resetpulse 194. The application of each transfer trigger pulse 198 causes anamount of photogenerated charge stored in photodiode PD3 represented bythe corresponding blackened region 260 to be transferred to a respectivestage of a corresponding vertical shift register, and the transferredcharge is accumulated therein for the duration of the exposure pulse193. Because transfer trigger pulses 198 occur at half the frequency ofthe transfer trigger pulses 195, the photogenerated charge accumulatedin the vertical shift register stage receiving photogenerated chargefrom photodiode PD3 over the exposure duration will be half of thatwhich is accumulated in the vertical shift register stage that receivesphotogenerated charge from photodiode PD4 over the same exposureduration. Operated in this manner, the effective sensitivity level ofphotodiode PD3 will be half that of photodiode PD4.

Transfer trigger pulses 261, which have the same width as the transfertrigger pulses 198 and 195, are applied to the transfer gate electrodeassociated with photodiode PD2. The phase of the transfer trigger pulses261 is such that each transfer trigger pulse 261 immediately precedesthe next reset pulse 194. The application of each transfer trigger pulse261 causes the transfer of an amount of photogenerated charge stored inphotodiode PD2 represented by the corresponding blackened region 263 toa respective stage of a corresponding vertical shift register, and thetransferred charge is accumulated therein for the duration of theexposure pulse 193. Because the frequency of the transfer trigger pulses261 is half that of the transfer trigger pulses 198 and one-quarter ofthat of the transfer trigger pulses 195, the total amount ofphotogenerated charge accumulated in the shift register stage thatreceives stored photogenerated charge from photodiode PD2 over theduration of the exposure pulse 193 will be half of that which isaccumulated in the vertical shift register stage that receives storedphotogenerated charge from photodiode PD3 and one-quarter of that whichis accumulated in the shift register stage that receives storedphotogenerated charge from photodiode PD4 over the same exposureduration. Consequently, the effective sensitivity level of photodiodePD2 is half of the effective sensitivity level of photodiode PD3 andone-quarter of the effective sensitivity level of photodiode PD4.

Transfer trigger pulses 264, which have the same width as transfertrigger pulses 261, 198 and 195, are applied to the transfer gateelectrode associated with photodiode PD1 at half the frequency of thatof transfer trigger pulses 261. The phase of the transfer trigger pulses264 is such that each transfer trigger pulse 264 immediately precedesthe next reset pulse 194. The application of each transfer trigger pulse264 causes an amount of photogenerated charge stored in photodiode PD1represented by the corresponding blackened region 266 to be transferredto a respective stage of a corresponding vertical shift register, andthe transferred charge is accumulated therein for the duration of theexposure pulse 193. Because the frequency of the transfer trigger pulses264 is half of that of transfer trigger pulses 261, one-quarter of thatof transfer trigger pulses 198 and one-eighth of that of transfertrigger pulses 195, the total photogenerated charge accumulated in thevertical shift register stage receiving stored photogenerated chargefrom photodiode PD1 over the duration of the exposure pulse 193 isone-half of that which is accumulated in the shift register stagereceiving stored photogenerated charge from photodiode PD2, one-quarterof that which is accumulated in the shift register stage receivingstored photogenerated charge from photodiode PD3 and one-eighth of thatwhich is accumulated in the vertical shift register stage which receivesstored photogenerated charge from photodiode PD4 over the same exposureduration. Consequently, the effective sensitivity level of photodiodePD1 is one-half of that of photodiode PD2, one-quarter of thatphotodiode PD3 and one-eighth of that photodiode PD4. Accordingly,different effective sensitivity levels may be obtained for thephotodiodes of the CCD image sensor by applying respective transfertrigger pulses to their associated transfer gate electrodes at differentfrequencies in the manner described.

Because photogenerated charge is accumulated in small amounts inrespective vertical shift register stages upon the application of eachtransfer trigger pulse and the transfer trigger pulses are uniformlydistributed over the exposure period, movement in the scene will beintegrated in the same manner in the vertical shift register stagesaccumulating stored photogenerated charge from respective photodiodes.Thus, inconsistency of motion blur in the output images is suppressed byusing the foregoing technique for obtaining a predetermined spatiallyvarying sensitivity pattern for the CCD image sensor.

Image Processing Part

Returning to the dataflow diagram of FIG. 2, the image processing part200 according to an exemplary embodiment of the present inventioncomprises a captured image memory 20, an off-grid estimator 22, anon-grid resampler 23 and an output image memory 21. The captured imagememory 20, which is physically implemented, for example, as part of theframe memory 12 in the block diagram of FIG. 3, stores in digitized formthe image captured by the image sensing part 1 of FIG. 1. The off-gridestimator 22 reads the captured image brightness values in the capturedimage memory 20 and computes brightness values at respective off-gridpositions, the off-grid positions being positions which are not on theimage (pixel position) array. The on-grid resampler 23 receives thebrightness values at respective off-grid positions and computesresampled brightness values at respective pixel positions by resamplingthe off-grid brightness values at respective off-grid positions. Thecomputed resampled brightness values at respective pixel positions arestored in the output image memory 21 as pixel brightness values of arelatively high dynamic range image of the scene.

Referring to FIG. 15, there is shown an exemplary arrangement 1500 ofpixel positions represented by rectangles 140 located in a pixelposition array defined by respective intersections of a plurality ofregularly spaced pixel rows 267 and a plurality of regularly spacedpixel columns 268 orthogonal to the pixel rows 267. The off-gridpositions are generally located at respective interstices of the pixelposition array form an off-grid position array. In the arrangement ofFIG. 15, the off-grid positions, which are represented by black dots141, form an off-grid position array defined by respective intersectionsof a plurality of regularly spaced intermediate rows 269 and a pluralityof regularly spaced intermediate columns 270 orthogonal to theintermediate rows 269. Each one of the intermediate rows 269 is parallelto the pixel rows 267 and extends medially between a respective adjacentpair of the pixel rows 267. Each one of the intermediate columns 270 isparallel to the pixel columns 268 and extends medially between arespective adjacent pair of the pixel columns 267. For convenientreference of the pixel positions and off-grid positions, the bottom leftcorner of the captured image is defined as the origin of a coordinatesystem (0,0) with the pixel position nearest the origin havingcoordinates (0.5, 0.5) and the off-grid position nearest the originhaving coordinates (1,1). The bottom right corner of the captured imageis defined as having coordinates (xSize, 0), where xSize is thedimension of the captured image in the direction (x direction) of thepixel rows 267, and the upper left corner of the image is defined ashaving coordinates (0, ySize), where ySize is the dimension of thecaptured image in the direction (y direction) of the pixel columns 268.

Turning to FIG. 16, there is shown a dataflow diagram 1600 of theoff-grid estimator (22 in the diagram of FIG. 2) according to anexemplary embodiment of the present invention. For this embodiment thelight-sensing elements of the image sensor have four differentsensitivity levels. The off-grid estimator 1600 has four resamplers 28,29, 30 and 31, each corresponding to a respective one of the foursensitivity levels, an off-grid position generator 32 for providing theoff-grid positions of the off-grid position array as defined by thediagram of FIG. 15, and a sensitivity pattern data memory 34 for storingdata indicative of the sensitivity level corresponding to each one ofthe pixel positions. In addition, the off-grid estimator of 1600includes weighting factor memories 24, 25, 26 and 27 for storingweighting factors W₁, W₂, W₃ and W₄, respectively, each weighting factorcorresponding to a respective one of the sensitivity levels of thelight-sensing elements, and multipliers 271, 272, 273 and 274, each ofwhich receive the output of a respective one of resamplers 28, 29, 30 or31 and weighting factors W₁, W₂, W₃, or W₄ corresponding to the samesensitivity level as the respective one of the resamplers, and providingthe product of the received resampler output and weighting factor. Theoff-grid estimator 1600 also includes an adder 275 which receives theoutputs of multipliers 271, 272, 273 and 274 and provides a sum of themultiplier outputs. Furthermore, the off-grid estimator 1600 has amapper 33, which receives the output of the adder 275, a lookup tablememory 35 associated with the mapper 33. The lookup table memory 35,which stores data representing the inverse of a sensitivity weightedcombined response function of light-sensing elements having differentones of the plurality of sensitivity levels, is used by the mapper 33 tocompensate the output of the adder 275 for each off-grid positionreceived from the off-grid position generator 32 by the inverse of thecombined response function to derive an estimated off-grid brightnessvalue at the received off-grid position. The off-grid estimator 1600also has an off-grid brightness value memory 36 for receiving the outputof the mapper 33.

Referring again to FIG. 3, the sensitivity pattern memory 34 in FIG. 16is physically implemented, for example, in the ROM 14 and stores thepredetermined spatially varying sensitivity pattern of the CCD imagesensor 9 in the form of a respective sensitivity level for each one ofthe pixel positions. The lookup table memory 35 in. FIG. 16 is alsophysically implemented, for example, in the ROM 14 and stores a lookuptable for compensating the output of the adder 275 at each one of theoff-grid positions by the inverse of a sensitivity weighted combinedresponse function of the light-sensing elements of the image sensor. Theoff-grid brightness value memory 36 in FIG. 16 is physicallyimplemented, for example, in the frame memory 12 and stores estimatedoff-grid brightness values at respective off-grid positions as providedby the mapper 33 in FIG. 16.

Turning back to FIG. 16, the captured image memory 20 stores capturedimage brightness values at respective pixel positions defined inaccordance with the diagram of FIG. 15. The brightness value at eachpixel position is produced, for example, by a correspondinglight-sensing element of the CCD image sensor 9 of the block diagram ofFIG. 3. Because each light-sensing element has one of the foursensitivity levels and produces a brightness value at a correspondingpixel position, each pixel position can be considered as having acorresponding sensitivity level. Each one of the resamplers 28, 29, 30and 31 receives an off-grid position from the off-grid positiongenerator 32 and sensitivity pattern data from the sensitivity patternmemory 34, and derives a sensitivity level off-grid brightness value forits corresponding sensitivity level at the received off-grid position byresampling the captured image brightness values in the captured imagememory 20 at pixel positions corresponding to that sensitivity level.The details of the resampling computation are described hereinbelow.After each one of the resamplers 28, 29, 30 and 31 derives a respectivesensitivity level off-grid brightness value for its correspondingsensitivity level at an off-grid position received from the off-gridposition generator 32, each one of the derived sensitivity leveloff-grid brightness values is multiplied by a respective one ofweighting factors W₁, W₂, W₃, or W₄ corresponding to the samesensitivity level as the deriving resampler in one of the multipliers271, 272, 273 and 274. The sensitivity level off-grid brightness valuesderived by the four resamplers 28, 29, 30 and 31 at the off-gridposition received from off-grid position generator 32 after scaling byrespective weighting factors W₁, W₂, W₃, or W₄ are then accumulated bythe adder 275 to derive a weighted sum sensitivity level off-gridbrightness value at the received off-grid position, which is provided tothe mapper 33. The mapper 33 rounds off the weighted sum sensitivitylevel off-grid brightness value from the adder 275 to obtain an integerindex for the lookup table memory 35. A combined response functioncompensated brightness value corresponding to the index is retrieved bythe mapper 33 from the lookup table memory 35 and provided to theoff-grid brightness value memory 36 as the estimated off-grid brightnessvalue at the off-grid position received from the off-grid generator 32.Since the off-grid position generator 32 sequentially generates off-gridpositions from (1,1) to (xSize−1, ySize−1), an off-grid brightness valuepattern with dimensions xSize−1 and ySize−1 is generated and stored inthe off-grid brightness value memory 36.

Referring to FIG. 17, there is shown a flow diagram of the computationprocess that may be carried out by the off-grid estimator 1600 of FIG.16. First, loop 201 is repeated for each off-grid position (x, y)provided by the off-grid position generator 32 of FIG. 16 from (1, 1) to(xSize−1, ySize−1), where x and y are the coordinates of the off-gridposition and xSize and ySize are the dimensions of the captured image inthe x and y directions, respectively. In the loop 201, steps 202, loop203 and step 206 are carried out. In step 202 a value s is initializedto 0. In loop 203 steps 204 and 205 are repeated for each i-thsensitivity level. In the present embodiment, the number of sensitivitylevels is 4, and, therefore, loop 203 is repeated four times. In step204, a sensitivity level off-grid brightness value p_(i) for the i-thsensitivity level at off-grid position (x,y) is computed from thecaptured image brightness values IC at respective pixel positions storedin the captured image memory 20 of FIG. 16 by using a resamplingfunction Resample_(i)(I_(c), S,x,y), where S represents the sensitivitypattern data from the sensitivity pattern memory 34 in the diagram ofFIG. 16. The details of the resampling function are describedhereinbelow. In step 205, the computed sensitivity level off-gridbrightness value for the i-th sensitivity level p_(i) is multiplied by apredetermined weighting factor W_(i) corresponding to the i-thsensitivity level, and the product is added to the value s. After loop203 is repeated for each sensitivity level, step 206 is performed. Instep 206 estimated off-grid brightness value B at off-grid position(x,y) is computed by the function Map (LUT,s) using the mapper 33 andthe lookup table memory 35 in the diagram of FIG. 16. The details ofthis function are described hereinbelow. After step 206 is finished, thecurrent repetition of loop 201 is completed. After loop 201 is repeatedfor each off-grid position received from the off-grid position generator32 in FIG. 16, the computation process represented by the flow chart1700 terminates.

Turning to FIG. 18, there is shown a flow diagram 1800 describing thecomputation of the function Resample_(i)(I_(c), S,x,y) used in thecomputation process described by the flow diagram 1700 of FIG. 17. Thecomputation described by the flow diagram 1800 is performed by the i-thresampler in FIG. 16 corresponding to the i-th sensitivity level. Instep 207 an image I_(i) having brightness values only at pixel positionscorresponding to the i-th sensitivity level is generated by masking thecaptured image I_(c) in accordance with the sensitivity pattern data S.In step 208 an off-grid brightness value p at a pixel position of themasked image I_(i) which is nearest to the off-grid position (x,y) isdetermined. Finally, in step 209, the brightness value p is provided asthe sensitivity level off-grid brightness value for the i-th sensitivitylevel at off-grid position (x, y), and the computation processrepresented by flow diagram 1800 terminates.

Turning now to FIG. 19, there is shown a flow diagram 1900 for thecomputation of the function Map(LUT,s) used in the computation processdescribed by flow diagram 1700 of FIG. 17. The computation of flowdiagram 1900 is performed by the mapper 33 and associated lookup tablememory 35 in the diagram of FIG. 16. The computation consists of asequence of steps 210, 211 and 212. In step 210 an integer value index tis computed by rounding off s, as computed in loop 203 of the flowdiagram 1700 of FIG. 17. In step 211 a value p is retrieved from thelookup table memory 35 in FIG. 16 using the index t. In step 212, thevalue p corresponding to index t in the lookup table memory 35 isprovided as the output, and the process represented by the flow diagram1900 terminates. The lookup table data which is used to map the index tto the value p is prepared in advance and stored in the lookup tablememory 35 in FIG. 16. As indicated above, the lookup table memory 35 isimplemented, for example, in the ROM 14 in FIG. 3.

Referring to FIG. 20, there is shown a graph 2000 of exemplarynormalized radiometric response functions of light-sensing elements withdifferent ones of the four sensitivity levels. The quantity E representsthe radiation intensity incident on the light-sensing element normalizedby the maximum radiation intensity that can be sensed by thelight-sensing element in order to create the graph 2000 of FIG. 20. Thequantity p represents the brightness value at the pixel positioncorresponding to the light-sensing element. Since the present embodimenthas four sensitivity levels, there are four response functions p₁ (E)142, p₂(E) 143, p₃(E) 144 and p₄(E) 145. Since all four responsefunctions are monotonic, a linear combination of these functions s(E)with positive weighting factors is also monotonic. The sensitivityweighted combined response function of the light-sensing elements havingthe four sensitivity levels s(E) may be expressed ass(E)=W ₁ p ₁(E)+W ₂ p ₂(E)+W ₃ p ₃(E)+W ₄ p4(E)  (1)The combining represented by equation (1) brings about a high dynamicrange for sensing the radiation intensity E because the responsefunctions at four sensitivity levels are combined. For example, p₁(E)has sensitivity to a bright range of E and p₄(E) has sensitivity to adark range of E, while p₂(E) and p₃(E) cover the mid range of E. Thecombining of equation (1) also results in an increase in the number ofgrey scale levels. If each of the response functions p₁(E), p₂(E), p₃(E)and p₄(E) has a grey scale of 256 levels, s(E) has a grey scaleequivalent to 1021 levels.

Turning to FIG. 21, there is shown a graph 2100 of an exemplary combinedresponse function s(E). To store lookup table data for the combinedresponse function s(E), the inverse of s(E), which is shown as curve 147in the graph 2200 of FIG. 22, can be used because the inverse of amonotonic function is also monotonic. If the inverse of s(E) isgenerated as a continuous function, a lookup table having any number oflevels can be digitized and stored. For example, a lookup table derivedfrom the graph of FIG. 22 has 65,536 levels. Moreover, data representingany kind of response function can be stored in a lookup table. Forexample, FIG. 23 shows a graph 2300 of the inverse of a responsefunction in which a gamma characteristic is included. Inclusion of thegamma characteristic may be necessary to provide a standardized videosignal output.

It will be apparent to those skilled in the art that the nearestneighbor sampling function carried out by the resamplers 28, 29, 30 and31 of FIG. 16 may be substituted with other well known samplingfunctions, such as bi-linear and bi-cubic sampling with interpolation.The weighting factors W₁, W₂, W₃ and W₄ can each be any positive value.In most instances, each of the weighting factors can be set to 1/N,where N is the number of sensitivity levels. In the embodiment of theoff-grid estimator shown in FIG. 16, if the weighting factors W₁, W₂, W₃and W₄ are all equal, the weighting factor memories 24, 25, 26 and 27,and the multipliers 271, 272, 273 and 274 may be eliminated, in whichcase the outputs of the resamplers 28, 29, 30 and 31 are provideddirectly to the adder 275. It is noted that the weighting factors may beselected to obtain a desired shape for the combined response functions(E) defined by equation (1) for purposes of creating the lookup tabledata stored in the lookup table memory 35 in the diagram of FIG. 4. Thelookup table data is derived by digitizing the inverse of the functions(E) as shown in the graph of FIG. 22. To minimize the digitizing error,the inverse of the function s(E) is digitized as uniformly as possible.Where the shape of the inverse of the function s(E) is not well suitedfor uniform digitization, the shape may be changed by changing theweighting factors.

Turning now to FIG. 24, there is shown a data flow diagram 2400 of theoff-grid estimator 22 of the diagram of FIG. 2 according to anotherexemplary embodiment of the present invention. The off-grid estimator ofFIG. 24 includes a pixel position generator 43 for sequentiallygenerating pixel positions from (0.5,0.5) to (xSize, ySize), asensitivity pattern memory 40 for storing a sensitivity levelcorresponding to each pixel position, a mapper 42 and associated lookuptable memory 39 for storing separate lookup table data representing theinverse of the response functions of light-sensing elements havingdifferent ones of the sensitivity levels, a mapped pixel positionbrightness value memory 41 for storing the brightness value atrespective pixel positions after compensation of each pixel positionbrightness value by the inverse of the response function of alight-sensing element having a sensitivity level corresponding to thepixel position, an off-grid position generator 45 for sequentiallygenerating off-grid positions from (1,1) to (xSize−1, ysize−1), apseudoinverse estimator for computing from the compensated pixelposition brightness values stored in the mapped pixel positionbrightness value memory 41 an estimated off-grid brightness value ateach off-grid position received from the off-grid position generator 45,and an off-grid brightness value memory 36 for storing estimatedoff-grid brightness values at respective off-grid positions computed bythe pseudoinverse estimator 44.

With reference to the block diagram of FIG. 3, the lookup table memory39 and the sensitivity pattern memory 40 are both physicallyimplemented, for example, in the ROM 14, and the mapped pixel positionbrightness value memory 41 is physically implemented, for example, inthe RAM 15.

Turning back to FIG. 24, the mapper 42 receives from the captured imagememory 20 a captured image brightness value at a pixel position receivedfrom the pixel position generator 43, and the mapper 42 applies to thecaptured image brightness value lookup table data stored in the lookuptable memory 39 for the sensitivity level corresponding to the receivedpixel position, as obtained from the sensitivity pattern memory 40, andderives a mapped brightness value at the received pixel position forstorage in the mapped pixel position brightness value memory 41. Themapped brightness value is the captured image brightness valuecompensated by the inverse of the response function of a light-sensingelement having the sensitivity of level corresponding to the receivedpixel position. Then the pseudoinverse estimator 44 computes abrightness value at a off-grid position provided by the off-gridposition generator 45 using the mapped brightness values in the mappedpixel position brightness value memory 41. The computed off-gridbrightness value at the off-grid position is then stored in an off-gridbrightness value storage 36.

Referring to FIG. 25, there is shown a flow diagram 2500 for acomputation process that may be carried out by the off-grid estimator2400 of the diagram of FIG. 24. First, loop 215 is repeated for eachpixel position (x,y) of the captured image from (0.5,0.5) to (xSize−0.5,ySize−0.5), as received from the pixel position generator 43 in FIG. 24.In loop 215 steps 216 and 217 are performed. In step 216, lookup tabledata LUT_(i) for the i-th sensitivity level is chosen by referring tothe sensitivity level i corresponding to received pixel position (x,y).The sensitivity level corresponding to received pixel position (x,y) isobtained from the sensitivity pattern data S(x,y) stored in thesensitivity pattern memory 40 of FIG. 24. In step 217, the lookup tabledata LUT_(i) is applied to a captured image brightness value I_(c) (x,y)at received pixel position (x,y) from the captured image memory 20. Thebrightness value compensated by the inverse of the response function ofa light-sensing element having sensitivity level i derived from thelookup table data LUT_(i) is referred to as the mapped brightness value,I_(m)(x,y), at received pixel position (x,y). After loop 215 is repeatedfor every pixel position as received from the pixel position generator43 in FIG. 24, loop 218 is repeated for each off-grid position (x′,y′)from (1, 1) to (xSize−1, ySize−1), as received from the off-gridposition generator 45 in FIG. 24. In loop 218, step 219, loop 220 andstep 223 are carried out. In step 219 a value s and a value w are eachinitialized to 0. After step 219, loop 220 is repeated for four pixelpositions (u,v) which are the four nearest neighbor pixel positions tothe received off-grid position (x′,y′). In loop 220, steps 221 and 222are performed. In step 221 a mapped brightness value I_(m)(u,v) at pixelposition (u,v), is compared with a low threshold value θ_(L) and a highthreshold value θ_(H). If I_(m)(u,v) is greater than θ_(L) and less thanθ_(H), step 222 is performed. Otherwise, step 222 is skipped and theprocess moves to the next repetition of loop 218. The threshold valuesθ_(L) and θ_(H) are predetermined constant values, which generallyrepresent the mapped noise level of a light-sensing element having thesensitivity level corresponding to pixel position (u,v), and thesaturation or near saturation brightness value of a light-sensingelement having the same sensitivity level, respectively. In step 222 amapped brightness value I_(m)(u,v), which is greater than θ_(L) and lessthan θ_(H), is added to s and 1 is added to w. After loop 220 isrepeated for each nearest neighbor pixel position, step 223 isperformed. In step 223, the brightness value B(x′,y′) at receivedoff-grid position (x′,y′) is computed by dividing the value s by thevalue w. After loop 218 is repeated for each off-grid position receivedfrom the off-grid position generator 45 in FIG. 24, the computationprocess represented by the flow diagram 2500 terminates.

Referring to FIG. 26, there is shown a flow diagram 2600 of analternative computation process which may be carried out by thepseudoinverse estimator 44 of the diagram of FIG. 24. The flow diagram2600 of FIG. 26, which implements a 4×4 cubic pseudoinverse computation,can replace the portion of the flow diagram 2500 of FIG. 25 thatincludes step 219, loop 220 and step 223. Referring to the flow diagram2600 of FIG. 26, in step 224 a value s and a value w are eachinitialized to 0. Then loop 225 is repeated for sixteen pixel positions(u,v) which are the sixteen nearest neighbor pixel positions to anoff-grid position (x′,y′) received from the off-grid position generator45 in FIG. 24. In loop 225, steps 226, 227 and 228 are performed. Instep 226, a mapped brightness value I_(m)(u,v) at nearest neighbor pixelposition (u,v) is compared with a low threshold value θ_(L) and with ahigh threshold value θ_(H). If I_(m)(u,v) is greater than θ_(L) and lessthan θ_(H), step 227 is performed. Otherwise, steps 227 and 228 areskipped and the process moves to the next iteration of the loop 225. Thethreshold values θ_(L) and θ_(H) are predetermined constant values,generally representing the mapped noise level of a light-sensing elementhaving the sensitivity level corresponding to pixel position (u,v), andthe saturation or near saturation brightness value of a light-sensingelement having the same sensitivity level, respectively. In step 227index values i and j are computed from the received off-grid position(x′,y′) and the nearest neighbor pixel position (u,v), where i and j areused to refer to values of a cubic interpolation kernel which is shownin FIG. 27. In step 228 s and w are each updated in accordance with theequations shown in the block representing step 228. After loop 225 isrepeated for each of the sixteen nearest neighbor pixel positions, step229 is performed. In step 229, a brightness value B at received off-gridposition (x′,y′) is computed as the value s divided by the value w. Thecomputation process represented by flow diagram 2600 is repeated for alloff-grid positions received from the off-grid position generator 45 inFIG. 24 from x′+1 to xSize−1 and y′+11 to ySize−1.

Turning to FIG. 28, there is shown a data flow diagram 2800 of theon-grid resampler 23 in the diagram of FIG. 2 according to an exemplaryembodiment of the present invention. The on-grid resampler 2800 includesa pixel position generator 38 for sequentially providing pixel positionsfrom (2.5,2.5) to (xSize−2.5, ySize−2.5), and an interpolator 37 whichcarries out a cubic interpolation operation that computes aninterpolated on-grid brightness value at a pixel position received fromthe pixel position generator 38 from the estimated off-grid brightnessvalues at 4×4 off-grid positions that are nearest neighbors to the pixelposition received from the pixel position generator 38. This isillustrated in FIG. 29 which shows a diagram 2900 of a local area ofpixel positions, which are represented as rectangles, and off-gridpositions, which are represented by black circles. When the brightnessvalue at a received pixel position (x,y) is computed, the 4×4 cubicinterpolation kernel is multiplied by 4×4 off-grid brightness valueswithin the local area from (x−2,y−2) to (x+2,y+2), which is shown as theshaded rectangle in the diagram of FIG. 29. An example of a 4×4 cubicinterpolation kernel 2700 is shown in FIG. 27. The pixel positiongenerator 38 in FIG. 28 sequentially generates pixel positions from(2.5, 2.5) to (xSize−2.5, ySize−2.5) and provides the pixel positions tothe interpolator 37. In this manner, resampled brightness values for animage having dimensions of (xSize−4,ySize−4) are generated and stored inthe output image memory 21 in FIG. 28.

Referring to FIG. 30, there is shown a flow diagram 3000 describingcomputation process that may be performed by the interpolator 37 in FIG.28. The computation process comprises loop 213, which includes step 214.Loop 213 is repeated for each pixel position (x, y) of the output imagefrom (2.5, 2.5) to (xSize−2.5, ySize−2.5), as received from the pixelposition generator 38 in FIG. 28, where x and y are the coordinates ofthe pixel position and xSize and ySize are the dimensions of thecaptured image in the x and y directions, respectively. In step 214 abrightness value of the output image I_(o)(x−2, y−2) at pixel position(x−2, y−2) is derived by computing the product of the brightness valuesat 4×4 nearest neighbor off-grid positions to the received pixelposition at (x,y) and a 4×4 cubic interpolation kernel φ, as shown bythe equation in the block representing step 214 in the flow diagram3000. The computed value is stored as the brightness value at thecorresponding pixel position of the output image I_(o)(x−2,y−2) in theoutput image memory 21 of the diagram in FIG. 28. After loop 213 isrepeated for each pixel position received from the pixel positiongenerator 38 in FIG. 28, the computational process represented by theflow diagram 3000 terminates.

While the interpolator 37 of the diagram in FIG. 28 is described asperforming a cubic interpolation, it is to be understood that anyinterpolation process which computes an interpolated value from twodimensional uniform grid data may be substituted for the cubicinterpolation. For example, other cubic kernels, Gaussian kernels,bi-linear interpolation or larger interpolation kernels can be used.Furthermore, shifting the pixel positions to coincide with the off-gridpositions and taking the brightness values at the off-grid positions asthe brightness values at respective coincident pixel positions can beused to derive resampled brightness values at the pixel positions.Examples of bi-cubic B-spline 3100, bi-linear 3200, 6×6 two dimensionalGaussian 3300 and a simple shift filter kernel 3400 are shown in FIGS.31, 32, 33 and 34, respectively. Each of these filter kernels may besubstituted as φ_(ij) in the flow diagram 3000 of FIG. 30. It is notedthat the range of values for i and j in the flow diagram 3000 of FIG. 30changes according to the size of the filter kernel. For example, whenthe 6×6 Gaussian filter 3300 of FIG. 33 is used, both i and j take onvalues from 0 to 5.

In the embodiments of the present invention described thus far, thelight-sensing elements of the image sensor have four differentsensitivity levels. It will be understood that the present invention maybe readily applied to embodiments where the light-sensing elements ofthe image sensor have a different number of sensitivity levels with onlyminor alterations to the four sensitivity level embodiments describedhereinabove. For example, the light-sensing elements of the image sensormay have only two sensitivity levels, in which case the modifications tothe embodiments are now described.

In the image sensing part where the spatially varying sensitivitypattern is defined by a mask, the mask will have cells with twodifferent transparencies, namely a dark cell and a bright cell. Forexample, the mask may have a checkerboard pattern in which adjacentcells in a row and adjacent cells in a column will have different onesof the two transparencies. An example of such a mask 3500 is depicted inFIG. 35, in which bright cells are designated by reference numeral 148and dark cells are designated by reference numeral 149.

In the image processing part, the off-grid position generators 32 in thediagram of FIGS. 16 and 45 in the diagram of FIG. 24 may generate thesame off-grid positions as defined in FIG. 15 for the two sensitivitylevel case. However, alternative off-grid positions may be defined whenthere are only two sensitivity levels.

Referring to FIG. 36, there is shown a diagram 3600 of an alternativeoff-grid position pattern for the two sensitivity level case. In FIG.36, pixel positions are represented by squares 150 and off-gridpositions are represented by black circles 151. Here the off-gridpositions 151 are again located at respective interstices of the pixelposition array. However, the interstices in the diagram of FIG. 36 arelocated at respective intersections of the pixel rows 267 and theintermediate columns 270, and at respective intersections of the pixelcolumns 268 and the intermediate rows 269. As in the diagram of FIG. 15,the pixel rows 267 and the pixel columns 268 are regularly spaced andmutually orthogonal, and each intermediate row 269 is parallel to thepixel rows 267 and extends medially between a respective adjacent pairof pixel rows 267, while each intermediate column 270 is parallel to thepixel columns 268 and extends medially between a respective pair ofpixel columns 268. It may be noted in the diagram of FIG. 36 that linesof adjacent off-grid positions are rotated by 45 degrees with respect tolines of adjacent pixel positions.

Turning to FIG. 37, there is shown a data flow diagram 3700 of theembodiment of the off-grid estimator of FIG. 16 modified for twosensitivity levels. The differences from the four sensitivity level caseare in the number of resamplers used, in the number of weighting factormemories and in the number of multipliers. The position generator 32 ismodified to generate the off-grid positions defined in FIG. 36. Thesensitivity pattern memory 34 stores a predetermined spatially varyingsensitivity pattern specifying one of the two sensitivity levelscorresponding to each pixel position.

Referring to FIG. 38, there is shown a flow diagram 3800 of thecomputation process that may be performed by the off-grid estimatorrepresented by the data flow diagram 3700 of FIG. 37. First, loop 230 isrepeated for each off-grid position (x, y) from (0.5,1) to (xSize−0.5,ySize−1), where x and y are coordinates of the off-grid position andxSize and ySize are the dimensions of the captured image in the xdirection and the y direction, respectively. With reference to thediagram 3600 of FIG. 36, the off-grid positions for which Loop 230 isrepeated are located at respective intersections of the intermediaterows 269 and the pixel columns 268. In loop 230, step 231, loop 232 andstep 235 are carried out. In step 231 a value s is initialized to 0. Inloop 232, steps 233 and 234 are repeated for each i-th sensitivitylevel. In the present embodiment the number of sensitivity levels istwo. In step 233 a sensitivity level off-grid brightness value p_(i) forthe i-th sensitivity level at an off-grid position (x,y) received fromthe off-grid position generator 32 in FIG. 37 is computed from thecaptured image brightness values at respective pixel positions of thecaptured image I_(c) by the resampling functionResample_(i)(I_(c),S,x,y) of the i-th resampler, where S represents thesensitivity pattern data from the sensitivity pattern memory 34 in FIG.37. The computation of this function is the same as in the foursensitivity level case. In step 234 the sensitivity level off-gridbrightness value p_(i) for the i-th sensitivity level is multiplied by apredetermined weighting factor W_(i) for the same sensitivity level, andthe product is added to s. After loop 232 is repeated for the twosensitivity levels, step 235 is performed. In step 235, brightness valueB at the off-grid position (x,y) received from the off-grid positiongenerator 32 in FIG. 37 is computed by applying the function Map(LUT,s)using the mapper 33 and associated lookup table memory 35 in FIG. 37.The application of this function is the same as in the four sensitivitylevel case. After step 235 is completed, the current repetition of loop230 is completed. After loop 230 is repeated for all off-grid positionslocated at respective intersections of the intermediate rows 269 and thepixel columns 268, the computation process proceeds to loop 236. Thecomputation process of loop 236 is identical to that of loop 230 exceptthat loop 236 is repeated for each off-grid position received from thisoff-grid position generator 32 in FIG. 37 from (1,0.5) to(xSize−1,ySize−0.5). Referring again to the diagram 3600 of FIG. 36, theoff-grid positions for which loop 236 is repeated are located atrespective intersections of the pixel rows 267 and the intermediatecolumns 270. After loop 236 is repeated for all such off-grid positions,the computation process represented by the flow diagram 3800 terminates.

The embodiment of the off-grid estimator represented by the data flowdiagram 2400 in FIG. 24 is also readily modified for two sensitivitylevels. Turning to FIG. 39, there is shown a flow diagram 3900describing a computation process that may be carried out by the off-gridestimator represented by data flow diagram 2400 of FIG. 24 for the twosensitivity level case. The first loop 256 is repeated for each pixelposition (x, y) of the captured image received from the pixel positiongenerator 43 in FIG. 24 from (0.5, 0.5) to (xSize−0.5, ySize−0.5). Inloop 256, steps 257 and 258 are performed. In step 257 lookup table dataLUT_(i) for the i-th sensitivity level is selected by referring to thesensitivity pattern data S(x, y) stored in the sensitivity patternmemory 40 for the sensitivity level corresponding to pixel position (x,y) received from the pixel position generator 43. In step 258 the lookuptable data LUT_(i) is applied to the captured image brightness valueI_(c)(x, y) at received pixel position (x, y), and the resultingcompensated brightness value at received pixel position (x, y) isdesignated the mapped brightness value I_(m)(x, y) at that pixelposition. After loop 256 is repeated for all pixel positions of thecaptured image received from pixel position generator 43, thecomputation process proceeds to loop 259. Loop 259 is repeated for eachoff-grid position (x′,y′) from (0.5, 1) to (xSize−0.5, ySize−1) receivedfrom the off-grid position generator 45 in FIG. 24. With reference tothe diagram 3600 in FIG. 36, the off-grid positions for which loop 259is repeated are located at respective intersections of the intermediaterows 269 and the pixel columns 268. In loop 259, step 260 is carriedout. In step 260 a function Pseudoinverse(I_(m), x′, y′) is evaluated toobtain the brightness value B(x′, y′) at received off-grid position (x′,y′). The details of evaluating the function Pseudoinverse (I_(m), x′,y′) will be further explained hereinbelow. After loop 259 is repeatedfor each off-grid position located at respective intersections of theintermediate rows 269 and the pixel columns 268, as received fromoff-grid position generator 45, the computation process proceeds to loop261, which is repeated for each off-grid position from (1, 0.5) to(xSize−1, ySize−0.5). With reference to diagram 3600 of FIG. 36, theoff-grid positions for which loop 261 is repeated are located atrespective intersections of the pixel rows 267 and the intermediatecolumns 270. Otherwise, the computation process of loop 261 is identicalto that of loop 259. After loop 259 is repeated for each of the off-gridpositions located at respective intersections of the pixel rows 267 andthe intermediate columns 270, as received from off-grid positiongenerator 45, the computation process represented by flow diagram 3900terminates.

Referring to FIG. 40, there is shown a flow diagram 4000 describing thecomputation process for the function Pseudoinverse(I_(m), x′, y′), whichcarries out a 4×4 rotated cubic pseudoinverse operation. First, in step263 a value s and a value w are each initialized to 0. After step 263,loop 264 is repeated for 16 pixel positions (u, v) which are nearestneighbors to an off-grid position (x′, y′) received from the off-gridposition generator 45 by incrementing indices i and j, which have valuesfrom 0 to 3, in each repetition of loop 264. In loop 264, steps 265,266, 267 and 268 are carried out. In step 265, coordinates u and v arecomputed from the coordinates of the current received off-grid position(x′, y′) and indices i and j using the relations shown in the blockrepresenting step 265 in the flow diagram 4000. In step 266 adetermination is made as to whether the coordinates u and v are those ofa pixel position. If coordinates u and v are those of a pixel position,the mapped brightness value I_(m)(u, v) at that pixel position iscompared with a low threshold value θ_(L) and a high threshold valueθ_(H) in step 267. Otherwise, steps 267 and 268 are skipped and thecomputation process moves to the next repetition of loop 264. If in step267 the mapped brightness value I_(m)(u, v) is determined to be greaterthan the low threshold value θ_(L) and less than the high thresholdvalue θ_(H), then step 268 is performed. Otherwise, step 268 is skippedand the computation process moves to the next repetition of loop 264.The threshold values θ_(L) and θ_(H) are predetermined constants, whichgenerally represent the mapped noise threshold of a light-sensingelement of the image sensor having the sensitivity level correspondingto pixel position (u,v), and the saturation or near saturationbrightness value of a light-sensing element of the image sensor havingthe same sensitivity level, respectively. In step 268, the values s andw are updated in the accordance with the equations shown in the blockrepresenting step 268 in the flow diagram 4000. In these equations,φ_(mn) represents the value of a 45° rotated cubic interpolation kernelφ at indices m and n. An example of a 45° rotated cubic interpolationkernel is shown in diagram 4200 in FIG. 42. After loop 264 is repeatedfor all values of the indices i and j, step 269 is performed. In step269 the value of the function Resample(I_(m), x′, y′) is obtained bydividing the value s by the value w, and the computation processrepresented by the flow diagram 4000 is repeated for the next off-gridposition (x′,y′) received from the off-grid position generator 45 inFIG. 24.

The data flow diagram of the on-grid resampler for the two sensitivitylevel case is the same as that for the four sensitivity level cases, asshown in FIG. 28. However, there are some differences in the computationcarried out by the interpolator 37 in the diagram of FIG. 28. As in thefour sensitivity level case, the interpolator carries out a cubicinterpolation operation which computes an interpolated brightness valueat a pixel position received from pixel position generator 38 in FIG. 28using respective estimated off-grid brightness values at 4×4 off-gridpositions that are nearest neighbors to the received pixel position. Theshape of the interpolation filter kernel for two sensitivity levels is,however, different. FIG. 41 shows a diagram of a local area of pixelpositions represented by squares and off-grid positions represented byblack circles. When the brightness value at a received pixel position(x, y) is computed, a 45° rotated 4×4 cubic interpolation kernel isapplied to form a product with the brightness values at 4×4 off-gridpositions that are nearest neighbors to the received pixel position(x,y) within a diamond shaped area from (x−2, y−2) to (x+2, y+2), whichis shown as a shaded diamond shaped region in FIG. 41. An example of a45° rotated 4×4 cubic interpolation kernel 4200 is shown in FIG. 42.Since the filter kernel for two sensitivity levels is more compact thanthe kernel for four sensitivity levels, the embodiment for twosensitivity levels results in better spatial frequency characteristics.

Turning to FIG. 43, there is shown a flow diagram 4300 describing acomputation process that may be carried out by the embodiment of theon-grid resampler for two sensitivity levels. The computation processcomprises loop 242, which includes step 243. Loop 242 is repeated foreach pixel position (x, y) of the output image received from pixelposition generator 38 in FIG. 28 from position (2.5, 2.5) to position(xSize−2.5, ySize−2.5), where x and y are the coordinates of the pixelposition, and xSize and ySize are the dimensions of the captured imagein the x direction and the y direction, respectively. In step 243, abrightness value at a received pixel position (x,y) of the output imageI_(o)(x,y) is derived by computing the product of the brightness value Bat the 4×4 off-grid positions that are nearest neighbors to the receivedpixel position (x,y), and the 45° rotated 4×4 cubic interpolation kernelφ in accordance with the equation shown in the block representing step243 in the flow diagram 4300. The computed brightness value at thecorresponding pixel position of the output image I (x−2,y−2) is storedin the output image memory 21 in FIG. 28. After repetition of loop 242for every pixel position received from pixel position generator 38 inFIG. 28 is completed, the computation process represented by the flowdiagram 4300 terminates.

Referring back to the data flow diagram 200 of FIG. 2, it is noted thatwhere the off-grid estimator 22 derives respective off-grid brightnessvalues at off-grid positions that lie in a regular array such as thatshown in FIG. 15 and FIG. 36, the regular array of off-grid brightnessvalues may be directly stored in the output image memory 21 as the pixelbrightness values of a high dynamic range output image, without furtherprocessing by an on-grid resampler 23. This is shown in the exemplaryembodiment of the present invention represented by the data flow diagramof FIG. 44, where the respective estimated off-grid brightness valuescomputed by the off-grid estimator 22 are provided directly to theoutput image memory 21 for subsequent display or further processing.Although an output image based directly on the estimated off-gridbrightness values will generally not have as high a quality as one thathas undergone resampling by the on-grid resampler 22, the quality may besufficient for some applications.

Turning to FIG. 45, there is shown a data flow diagram 4500 according toanother exemplary embodiment of the present invention, in which themapper 33 and associated lookup table memory 35 of the off-gridestimator of FIG. 16 are moved from the location shown in FIG. 16 toafter the interpolator 37 of the on-grid resampler of FIG. 28.Approximately the same result is obtained using the arrangement of FIG.45 as is obtained with the combination of the off-grid estimator of FIG.16 and the on-grid resampler of FIG. 28.

Turning to FIG. 46, there is shown a flow diagram 4600 of thecomputational process that may be carried out by the arrangement of FIG.45. First, loop 244 is repeated for each off-grid position (x′, y′)received from off-grid position generator 32 in FIG. 45 from (1, 1) to(xsize−1,ysize−1), where x′ and y′ are the coordinates of the off-gridpositions, and xSize and ySize are the dimensions of the captured imagein the x direction and y direction, respectively. In loop 244, step 245,loop 246 and step 249 are carried out. In step 245 a value s isinitialized to 0. In loop 246, steps 247 and 248 are repeated for eachi-th sensitivity level. In the present embodiment, the number ofsensitivity levels is 4. In step 247 a sensitivity level off-gridbrightness value p_(i) for the i-th sensitivity level at receivedoff-grid position (x′, y′) is derived from the captured image brightnessvalues I_(c) in the captured image memory 20 by the resampling functionResample_(i)(I_(c), S, x′, y′) of the i-th resampler, where S representsthe sensitivity pattern data from the sensitivity pattern memory 34 inFIG. 45. In step 248 the computed sensitivity level off-grid brightnessvalue for the i-th sensitivity level p_(i) is multiplied by apredetermined weighting factor W_(i) for the i-th sensitivity level. Theproduct of the multiplication is added to s. After loop 246 is repeatedfor each sensitivity level, step 249 is performed. In step 249 the values is assigned to the off-grid brightness value B at off-grid position(x′, y′) received from off-grid position generator 32 in FIG. 45. Afterstep 249 is performed, the current repetition of loop 244 is completed.After loop 244 is repeated for every off-grid position, as received fromoff-grid position generator 32, the computation proceeds to loop 250.Loop 250 is repeated for each pixel position (x, y) from (2.5, 2.5) to(xSize−2.5, ySize−2.5), as received from pixel position generator 38 inFIG. 45. In loop 250, steps 251 and 252 are performed. In step 251 aninterpolated on-grid brightness value I at pixel position (x,y) isderived by computing the product of the brightness values at 4×4off-grid positions that are nearest neighbors to a received pixelposition (x, y) and a 4×4 cubic interpolation kernel φ using theequation in the block representing step 251 in the flow diagram 4600. Instep 252, the function MAP(LUT,I) is evaluated to obtain the pixelbrightness value of the output image I_(o)(x−2, y−2) at pixel position(x−2,y−2). The computation process for evaluating the functionMAP(LUT,I) is described by the flow diagram 1900 in FIG. 19. After loop250 is repeated for every pixel position received from pixel positiongenerator 38 in FIG. 45, the computation process represented by the flowdiagram for 4600 terminates.

Referring now to FIG. 47, there is shown a data flow diagram 4700 ofanother exemplary embodiment of the present invention, which uses anon-grid interpolation filter 46 to implement the image processing part 2of FIG. 1. Comparing the diagram in FIG. 47 with the diagram in FIG. 45,it is noted that the resamplers 28, 29, 30 and 31, the weighting factormemories 24, 25, 26 and 27, the sensitivity pattern memory 34, theoff-grid brightness value memory 36, the interpolator 37, the off-gridposition generator 32 and the pixel position generator 38 are replacedby an on-grid interpolation filter 46 and a pixel position generator 47.The derivation of the filter is now explained.

In accordance with the present invention, off-grid brightness valuesB(x,y) are obtained by applying an off-grid estimating functionƒ_(off)(*) to the captured image brightness values I_(c)(x,y), i.e.:B(x,y)=ƒ_(off)(I _(c)(x,y)).  (2)In the embodiment of the off-grid estimator represented by FIGS. 16-19,off-grid brightness values dependent on a monotonic light-sensingelement response function s(E) are first derived, and then responsefunction independent off-grid brightness values are recovered byapplying a lookup table for the inverse of the response function s(E),i.e.:B(x,y)=s ¹({tilde over (s)}(I _(c)(x,y))),  (3)where {tilde over (s)}(*) is a function for obtaining sensitivity leveldependent off-grid brightness values by using a weighted sum of responsefunctions of light-sensing elements having different sensitivity levelssuch as that computed using equation (1).

To obtain the output image, an on-grid resampling function ƒ_(on)(*) isused to compute brightness values at pixel positions by a conventionalresampling function r(*). Therefore, the output image can be expressedas

$\begin{matrix}\begin{matrix}{I_{({x,y})} = {f_{on}\left( {f_{off}\left( {I_{c}\left( {x,y} \right)} \right)} \right)}} \\{= {r\left( {{s^{- 1}\left( {\overset{\sim}{s}\left( {I_{c}\left( {x,y} \right)} \right)} \right)}.} \right.}}\end{matrix} & (4)\end{matrix}$It may be assumed that the same continuous function can be constructedfrom different resampled data sequences if the original continuousfunction is sufficiently smooth. Based on this assumption, Ĩ(x,y) isequivalent to I_(o)(x,y), whereĨ _(o)(x,y)=s ⁻¹(r({tilde over (s)}(I _(c)(x,y))).  (5)Although Ĩ_(o) and I_(o) are not equivalent when {tilde over(s)}(I_(c)(x,y)) is not smooth, Ĩ_(o)(x,y) can nevertheless be used asan approximation of I_(o)(x,y).

The functions {tilde over (s)}(*) and r(*) may be combined into a singlefilter function because both functions are linear and invariant to phaseshift. Therefore the embodiment represented by the diagram of FIG. 47uses an on-grid interpolation filter that combines {tilde over (s)}(*)and r(*). The filter is derived by the convolution of {tilde over(s)}(*) and r(*). An example of the kernel 4900 of such a filter thatcombines {tilde over (s)}(*) and r(*) is shown in FIG. 49. The filterkernel 4800 is derived by using simple averaging of four nearestneighbor pixel position brightness value resampling described for theembodiment of the off-grid estimator of FIGS. 16-19 as {tilde over(s)}(*), and conventional cubic interpolation filtering as r(*).

Referring to FIG. 48, there is shown a flow diagram 4800 describing thecomputation process carried out by the embodiment of the imageprocessing part represented by the data flow diagram 4700 in FIG. 47.First, loop 253 is repeated for each pixel position (x, y) of the outputimage, as received from pixel position generator 47 in FIG. 47, from(2.5,2.5) to (xSize−2.5,ySize−2.5), where x and y are the coordinates ofthe pixel position, and xSize and ySize are the dimensions of thecaptured image in the x and y directions, respectively. In loop 253,steps 254 and 255 are carried out. In step 254 an interpolated on-gridbrightness value I is computed by multiplying the brightness values ofthe captured image I_(c) at 5×5 nearest neighbor pixel positions to areceived pixel position (x,y) by a 5×5 filter kernel φ. In step 255 thelookup table data, which is stored in the lookup table memory 35 in FIG.47, is applied by the mapper 33 in FIG. 47 to the computed on-gridbrightness value I to obtain the brightness value of the output imageI_(o)(x−2,y−2) at pixel position (x−2,y−2). After loop 253 is repeatedfor every output image pixel position received from the pixel positiongenerator 47 in FIG. 47, the computation process represented by the flowdiagram 4800 terminates. Using the process represented by flow diagram4800, an output image with dimensions of xSize−4 and ySize−4 isobtained.

The present invention may be applied to linear image sensors havinglinear arrays of light-sensing elements that capture line images. Suchlinear image sensors, including CCD linear image sensors, are availablewith a greater number of light-sensing elements along a given directionthan two-dimensional image sensors. Accordingly, linear image sensorsmay be used to provide two-dimensional images of greater resolution inapplications where scanning of a scene with the linear image sensor isfeasible and sequentially captured line images of successive regularlyspaced linear regions of the scene are concatenated. Examples of theways that a linear image sensor may be used to scan a scene andsequentially capture line images that may be concatenated to for atwo-dimension image of the scene are illustrated in FIGS. 50 and 51.Referring to FIG. 50, there is shown an imaging system having a linearimage sensor 276 and a lens system 277 for projecting a line image of alinear region 310 of a scene having an object 278 onto the light-sensingelements (not shown) of the linear image sensor. The imaging system ismoved (translated and/or rotated) at a constant rate with respect to thescene as shown by the arrow indicating the motion of the imaging systemin the figure. Line images of the scene are captured at regularsequential intervals by applying a regular sequence of exposure signals(not shown) to the linear image sensor 276 during the motion. Thesequentially captured line images can be concatenated to form a singlehigh resolution two-dimensional image of the scene.

Turning to FIG. 51, there is shown an imaging system comprising a linearimage sensor 276 and a lens system 277. The imaging system is heldstationary with respect to a scene comprising a conveyor belt 279carrying objects 280 and 281 moving past the imaging system at aconstant speed. Line images of successive regularly spaced linearregions 311 of the scene are sequentially captured at regular intervalsby applying a regular sequence of exposure signals (not shown) to thelinear image sensor 276. The sequentially captured line images can againbe concatenated to form a two-dimensional image of a section of theconveyor belt 279 carrying objects 280 and 281.

In accordance with the present invention, the linear image sensor 276has a spatially varying sensitivity pattern. Referring to FIG. 52, thereis shown an illustrative linear mask 5200 having cells 282 and 283 withtwo different transparencies. Each cell of the mask 5200 is placed infront of a respective one of the light-sensing elements of the linearimage sensor so as to cause each light-sensing element of the sensor tohave a respective one of two sensitivity levels in accordance with thepredetermined sensitivity pattern of the linear array of light-sensingelements as defined by the mask 5200.

Each light-sensing element of the linear image sensor in response toincident light from the scene produces a captured image brightness valueat a corresponding one of a multiplicity of regularly spaced pixelpositions of a linear pixel position array, thereby capturing a lineimage. When successive line images captured by the linear image sensorare concatenated, there is obtained a two-dimensional array of pixelpositions located at respective intersections of a plurality ofregularly spaced pixel rows and a plurality of regularly spaced pixelcolumns orthogonal to the pixel rows, where each pixel position has arespective brightness value produced by a light-sensing element thatcorresponds to the same pixel position in each one of the concatenatedline images, and corresponds to a respective one of the two sensitivitylevels of the light-sensing elements in accordance with atwo-dimensional sensitivity pattern as defined illustratively by theconcatenated exposure mask of FIG. 53. It will be understood that thelight-sensing elements of the linear image sensor may have more than twosensitivity levels and that the sensitivity level of each light-sensingelement may be established by means other than a mask.

Referring back to FIG. 37, the captured image brightness values atrespective pixel positions of the concatenated two-dimensional capturedimage may be stored in captured image memory 20 and the spatiallyvarying sensitivity pattern as represented illustratively by FIG. 53 maybe stored in the sensitivity pattern memory 34. The off-grid brightnessvalues may be estimated in accordance with the data flow diagram shownin the figure, and the array of off-grid positions may be aconcatenation of the linear off-grid position array shown illustrativelyin FIG. 54, where each off-grid position 286 is located midway between arespective adjacent pair of pixel positions 285.

The data flow diagram of FIG. 24 and the computational processesdescribed by the flow diagrams 3900 and 4000 of FIGS. 25 and 30 withappropriate modifications to account for a different off-grid positionarray may also be used to estimate the off-grid position brightnessvalues from the concatenated two-dimensional captured image brightnessvalues in the captured image memory 20. The estimated off-gridbrightness values at the respective off-grid positions may be useddirectly as the pixel brightness values of an output image in accordancewith FIG. 44, or on-grid resampling of the estimated off-grid brightnessvalues in accordance with the data flow diagram of FIG. 28 and thecomputation process described by the flow diagram 3000 of FIG. 30 withappropriate modifications to account for a different off-grid array, maybe used to derive the resampled on-grid brightness values at respectivepixel positions of the output image, the resampled on-grid brightnessvalues being the pixel brightness values of the output image stored inthe output image memory 21.

Referring to FIGS. 45 and 46, the captured image brightness values atrespective pixel positions of the concatenated captured line imagesstored in the captured image memory 20 may be processed in accordancewith the data flow diagram 4500 of FIG. 45 using the computation processdescribed by the flow diagram 4600 of FIG. 46 to derive resampledbrightness values at respective pixel positions of the output image tobe stored in the output image memory 21. To carry out such processing,the data flow diagram 4500 and the computation flow diagram 4600 aremodified to take into account the use of an image sensor withlight-sensing elements having two sensitivity levels and the use of adifferent off-grid position array.

Referring to FIGS. 47 and 48, the captured image brightness values atrespective pixel positions of the concatenated captured line imagesstored in the captured image memory 20 may be processed in accordancewith the data flow diagram 4700 of FIG. 47 using the computation processdescribed by the flow diagram 4800 of FIG. 48 to derive compensatedinterpolated brightness values at respective pixel positions of theoutput image to be stored in the output image memory 21. To carry outsuch processing, the data flow diagram 4700 and the flow diagram 4800are modified to take into account the use of an image sensor withlight-sensing elements having two sensitivity levels. Aside from changesin the data stored in the lookup table 35 representing the inverse of acombined response function of light-sensing elements having the twosensitivity levels, these modifications will also result in a differentfilter kernel φ to be used in the computation process described by theflow diagram 4800.

As an alternative to processing concatenated captured line images, eachline image may be separately processed to derive a high dynamic rangeline image by estimating the brightness values at off-grid positions,which for a linear pixel position array are located between respectiveadjacent pairs of pixel positions. The processed line images may then beconcatenated to obtain a high dynamic range two-dimensional image.Separate processing of each line image is less computationally intensivein comparison to the processing of a two-dimensional image, andtherefore can be done in real time as the line images are captured.Turning to FIG. 54, there is shown an illustrative row of pixelpositions 285 represented by squares, and “off-grid” positions 286represented by circles, where each off-grid position 285 is locatedmidway between a respective adjacent pair of pixel positions 285.According to one exemplary embodiment, the brightness value at each oneof the off-grid positions 286 may be derived by computing a weightedaverage of the captured image brightness values at the pixel positions285 within a pre-defined neighborhood of the off-grid position. In thecase where the linear image sensor has light-sensing elements of twodifferent sensitivity levels, the captured image brightness values atall consecutive adjacent pairs of pixel positions are averaged to obtainthe estimated brightness value at the off-grid position between each ofthe consecutive adjacent pairs of pixel positions. In this manner, thebrightness value B(x′) at off-grid position x′ may be expressed asB(x′)=W ₁ I _(c)(x′−0.5)+W ₂ I _(c)(x′+0.5),  (6)where W₁ and W₂ are weighting factors assigned to the two differentsensitivity levels. The resulting off-grid brightness values represent aone-dimensional image with high dynamic range. The computed off-gridbrightness values are each compensated by the inverse of a combinedresponse function of the light-sensing elements. The combined responsefunction may be expressed ass(E)=W ₁ P ₁(E)+W ₂ P ₂(E)  (7)where P₁(E) is the radiometric response function of a light-sensingelement of the linear image sensor having one of the two sensitivitylevels, P₂(E) is the radiometric response function of a light-sensingelement of the linear image sensor having the other of the twosensitivity levels, and W₁ and W₂ are the weighting factors assigned tothe two different sensitivity levels. Compensation of the estimatedoff-grid brightness values by the inverse of the combined responsefunction of the light-sensing elements using a mapper and a lookup tablememory as in the two-dimensional case remaps the off-grid brightnessvalues to ones that are linearly related to scene radiance.

Referring to FIG. 55, there is shown a flow diagram 5500 describing acomputation process for deriving the estimated brightness values at theoff-grid positions 286 in FIG. 54. The computation process described bythe flow diagram 5500 is carried out by an off-grid estimatorrepresented by the data flow diagram 2400 of FIG. 24. Referring to FIG.24, the pixel position generator 43 provides the positions of the pixelposition 285 of the linear pixel position array shown in FIG. 54. Thesensitivity pattern memory 40 stores the sensitivity level correspondingto each pixel position as defined by the illustrative mask 5200 in FIG.52. The lookup table memory 39 stores data representing respectiveinverse response function data of light-sensing elements having the twosensitivity levels. The off-grid position generator 46 provides theoff-grid positions 286 of the off-grid position array shown in FIG. 54.

Turning back to FIG. 55, loop 287 is first repeated for each pixelposition x of the linear pixel position array from 0.5 to xSize−0.5 asreceived from the pixel position generator 43 in FIG. 24, where xSize isthe length of the line image. In loop 287, steps 288 and 289 areperformed. In step 288, lookup table data LUT_(i) for the i-thsensitivity level is chosen by referring to the sensitivity level icorresponding to received pixel position x. The sensitivity levelcorresponding to the received pixel position x is obtained fromsensitivity pattern data S(x) stored in the sensitivity pattern memory40 in FIG. 24. In step 289, the lookup table data LUT_(i) stored in thelookup table memory 39 in FIG. 24 is applied by the mapper 42 in FIG. 24to the captured image brightness value I_(c)(x) at received pixelposition x from the captured image memory 20 in FIG. 24. The brightnessvalue at received pixel position x compensated by the inverse of theresponse function of the light-sensing element having sensitivity leveli derived from the lookup table data LUT_(i) is designated by I_(m)(x).After loop 287 is repeated for every pixel position received from thepixel position generator 43 in FIG. 24, loop 290 is repeated for eachoff-grid position x′ from 2 to xSize−2 as received from the off-gridposition generator 45 in FIG. 24. In loop 290, step 291, loop 292 andstep 293 are performed. In step 291, a value p and a value q are eachinitialized to 0. After step 291, loop 292 is repeated for each pixelposition k that is one of a predetermined number of pixel positions thatare nearest neighbors to the received off-grid position x′. In thecomputation process described by the flow diagram 5500 of FIG. 55, thepredetermined number of nearest neighbor pixel positions is four and khas values from (x′−1.5) to (x′+1.5). In loop 292, steps 294 and 295 areperformed. In step 294, a compensated captured image brightness value atpixel position k, I_(m)(k), is compared with a low threshold value θ_(L)and a high threshold value θ_(H). If I_(m)(k) is greater than θ_(L) andless than θ_(H) step 295 is performed. Otherwise, loop 292 proceeds toits next repetition. In step 295, I_(m)(k) is multiplied by the value ofa 4×1 Gaussian interpolation kernel G(k−x′) at position (k−x′), and theproduct is added to the quantity p. In addition, the value of theGaussian interpolation kernel G(k−x′) is added to the quantity q. Afterloop 292 is repeated for each value of k, step 293 is performed. In step293, the off-grid brightness value B(x′) at the received off-gridposition x′ is computed by dividing the value p by the value q, and loop290 proceeds to the next off-grid position received from the off-gridposition generator 45 in FIG. 24. After loop 290 has been repeated foreach off-grid position x′ of the off-grid position array, thecomputation process described by flow diagram 5500 terminates. Thethreshold values θ_(L) and θ_(H) are predetermined constant values,which generally represent the noise level of a light-sensing elementhaving the sensitivity level corresponding to pixel position kcompensated by the inverse of the response function of the light-sensingelement, and the saturation or near saturation value of a light-sensingelement having the same sensitivity level, respectively. An example of a4×1 Gaussian kernel 5600 that may be used in the computation processdescribed by the flow diagram 5500 is shown in FIG. 55, assuming thatthe center of the Gaussian kernel 5600 is at the center of the 4×1 pixelposition array involved in the interpolation.

The respective estimated off-grid brightness values at the off-gridpositions may be used directly as the pixel brightness values of anoutput image and stored in the output image memory 36 in FIG. 24.Alternatively, the off-grid brightness values may be resampled using oneof a variety of interpolation techniques to obtain resampled pixelposition brightness values that are then stored in the output imagememory as pixel brightness values of the output image.

Referring back to FIG. 28, resampling of the estimated off-gridbrightness values to derive respective resampled brightness values atthe pixel positions may be in accordance with the data flow diagram of2800. An interpolator 37 receives pixel positions of the linear pixelposition array illustrated in FIG. 54 from the pixel position generator38. For each received pixel position the interpolator 37 derives fromthe estimated off-grid brightness values stored in the off-gridbrightness value memory 36 an interpolated brightness value at thereceived pixel position, which is then provided to the output imagememory 21. Turning to FIG. 57, there is shown a flow diagram 5700describing the computation process that may be performed by theinterpolator 37 in FIG. 28. The computation process comprises loop 297,which includes step 298. Loop 297 is repeated for each pixel position xof the output image from 3.5 to xSize−3.5, as received from the pixelposition generator 38 in FIG. 28, where xSize is the length of thecaptured line image. In step 298 a brightness value of the output imageI_(o)(x−3) is derived by computing the product of the brightness valuesat four nearest neighbor off-grid positions to the received pixelposition x, and a 4×1 cubic interpolation kernel ψ. The computed on-gridbrightness value I_(o)(x−3) at pixel position (x−3) is stored as thebrightness value at the corresponding pixel position of the output imagein the output image memory 21 of FIG. 28. After loop 297 is repeated foreach pixel position from 3.5 to xSize−3.5 received from the pixelposition generator 38 in FIG. 28, the computation process described bythe flow diagram 5700 terminates. An example of a cubic interpolationkernel ψ (k) 5800 that may be used in the computation process describedby the flow diagram 5700 is shown in FIG. 58.

Another embodiment of the present invention for processing captured lineimages to derive on-grid brightness values of a line output image isrepresented by the data flow diagram 4700 of FIG. 47. Referring to FIG.47, the captured image brightness values at respective pixel positionsof a linear pixel position array as illustrated in FIG. 54 are stored inthe captured image memory 20. An on-grid interpolation filter 46receives pixel positions of the linear pixel position array from a pixelposition generator 47. For each pixel position received from the pixelposition generator 47 the filter 46 interpolates the captured imagebrightness values in the captured image memory 20 to derive aninterpolated on-grid brightness value at the received pixel position.The interpolated on-grid brightness value is then compensated by theinverse of a combined response function of the light-sensing elements ofthe image sensor (not shown), and the compensated interpolated on-gridbrightness value is stored in the output image memory 21 as an outputline image brightness value at the pixel position received from thepixel position generator 47.

Referring to FIG. 59, there is shown a data flow diagram 5900representing a further embodiment of the image processing part 2 of thedata flow diagram of FIG. 1. When used for processing captured lineimages, the image processing part of FIG. 59 includes a first pixelposition generator 43 for sequentially generating pixel positions of thelinear pixel position array as illustrated in FIG. 54 from 0.5 toxSize−0.5, where xSize is the length of the line image. The imageprocessing part of FIG. 59 also includes a sensitivity pattern memory 40for storing the sensitivity level corresponding to each pixel positionof the linear pixel position array, a mapper 42 and associated lookuptable memory 39 for storing separate lookup table data representing theinverse of the response functions of light-sensing elements havingdifferent ones of the sensitivity levels, a mapped pixel positionbrightness value memory 41 for storing the captured image brightnessvalue at each pixel position after compensation by the inverse of theresponse function of a light-sensing element having a sensitivity levelcorresponding to the pixel position, a second pixel position generator47 for generating pixel positions of the pixel position array from 2.5to xSize−2.5, an on-grid interpolation filter 46 receiving pixelpositions from the second pixel position generator 43 and interpolatingthe compensated captured image brightness values at respective pixelpositions in the mapped pixel position brightness value memory 41 toderive an on-grid interpolated brightness value for each pixel positionreceived from the second pixel position generator 47. The on-gridinterpolated brightness values at respective pixel positions of thelinear pixel position array are stored in the output image memory 21 asthe pixel brightness values of an output line image. It is noted thatthe embodiment of the image processing part represented by the data flowdiagram 5900 of FIG. 59 may be used to process two-dimensional capturedimages with appropriate changes to take into account a two-dimensionalpixel position array, a two-dimensional sensitivity pattern and atwo-dimensional on-grid interpolation filter.

Turning to FIG. 60, there is shown a flow diagram 6000 describing acomputation process that may be carried out by the embodiment of theimage processing part represented by the data flow diagram 5900 of FIG.59 for processing a captured line image. First, loop 299 is repeated foreach pixel position x received from the first pixel position generator43 in FIG. 59 from 0.5 to xSize−0.5, where xSize is the length of thecaptured line image. In loop 299, steps 301 and 302 are performed. Instep 301 lookup table data LUT_(i) for the i-th sensitivity level storedin the lookup table memory 39 in FIG. 59 is selected by referring to thesensitivity pattern data S(x) stored in the sensitivity pattern memory40 in FIG. 59 for the sensitivity level corresponding to pixel positionx received from the first pixel position generator 43 in FIG. 59. Instep 302, the lookup table data LUT_(i) is applied to the captured imagebrightness value I_(c)(x) at the received pixel position x by the mapper42 in FIG. 59, and the resulting compensated captured image brightnessvalue at received pixel position x is designated the mapped brightnessvalue, I_(m)(x), at that pixel position. After loop 299 is repeated forall pixel positions of the pixel position array received from the firstpixel position generator 43, the computation process proceeds to loop303. Loop 303 is repeated for each pixel position x from 2.5 toxSize−2.5. In loop 303, step 304, loop 305 and step 306 are performed.In step 304, quantities p and q are each set to 0. After step 304, loop305 is repeated for five pixel positions k which are nearest neighborsto the pixel position x received from the second pixel positiongenerator 47 by incrementing the pixel position k in each repetition ofloop 305 from k=x−2 to k=x+2. In loop 305, steps 307, 308 and 309 arecarried out. In step 305, a mapped brightness value at pixel position k,I_(m)(k), is compared with a low threshold value θ_(L) and a highthreshold value θ_(H). If I_(m)(k) is greater than θ_(L) and less thanθ_(H), step 308 is performed. Otherwise, loop 305 proceeds to the nextvalue of k. In step 308, I_(m)(k) is multiplied by the value of a linearGaussian interpolation kernel G(k−x) at position (k−x), and the productis added to the quantity p. In addition, the value of the Gaussianinterpolation kernel G(k−x) at position (k−x) is added to the quantityq. After loop 305 is repeated for each value of k from k=x−2 to k=x+2,step 306 is performed. In step 306, the output image brightness valueI_(o)(x) at the pixel position x received from the second pixel positiongenerator 47 is computed by dividing the value p by the value q. Theloop 303 then proceeds to the next pixel position received from thesecond pixel position generator 47. After loop 303 has been repeated foreach pixel position x from x=2.5 to x=xSize−2.5, the computation processrepresented by flow diagram 6000 terminates. The threshold values θ_(L)and θ_(H) are predetermined constant values, which generally representthe noise level of a light-sensing element having the sensitivity levelcorresponding to pixel position k compensated by the inverse of theresponse function of that light-sensing element, and the saturation ornear saturation value of the light-sensing element having the samesensitivity level, respectively. An example of a 5×1 Gaussian kernelG(k), which may be used in the computation process described by the flowdiagram 6000 is shown in FIG. 61.

While the present invention has been particularly described withreference to exemplary embodiments thereof, it will be understood bythose skilled in the art that various modifications and alterations maybe made without departing of the spirit and the scope of the invention.For example, the image sensor need not be a charge coupled device, butmay be any image sensor having an array of light-sensing elements, suchas a CMOS image sensor. Accordingly, the disclosed embodiments of theinvention are considered merely illustrative, and the invention islimited in scope only as specified in the appended claims.

1. A system for obtaining a relatively high dynamic range image of ascene using a relatively low dynamic range image sensor adapted to beexposed to incident light from the scene for capturing an image thereofcomprising: an image sensor having a multiplicity of light-sensingelements in an array, each light-sensing element having a particular oneof a plurality of sensitivity levels to incident light in accordancewith a predetermined sensitivity pattern for the array of light-sensingelements and a respective response function, each light-sensing elementbeing responsive to incident light from the scene for producing acaptured image brightness value at a corresponding one of a multiplicityof pixel positions of a pixel position array, whereby each one of themultiplicity of pixel positions corresponds to a particular one of theplurality of sensitivity levels of the light-sensing elements; acaptured image memory for storing the captured image brightness valuesproduced by the light-sensing elements at corresponding ones of themultiplicity of pixel positions of the pixel position array; an off-gridestimator for deriving from the captured image brightness values in thecaptured image memory respective estimated off-grid brightness values ata multiplicity of off-grid positions located at respective intersticesof the pixel position array so as to form a regular off-grid positionarray, the off-grid estimator comprising: an off-grid position generatorfor providing the off-grid positions of the off-grid position array; asensitivity pattern memory for storing data indicative of thesensitivity level corresponding to each one of the pixel positions ofthe pixel position array; a plurality of sensitivity level off-gridbrightness value estimators, each corresponding to a respective one ofthe sensitivity levels, for receiving off-grid positions from theoff-grid position generator and sensitivity pattern data from thesensitivity pattern memory, each one of the plurality of sensitivitylevel off-grid brightness value estimators being responsive to anoff-grid position received from the off-grid position generator and thesensitivity pattern data from the sensitivity pattern memory forderiving from the captured image brightness values in the captured imagememory a respective sensitivity level off-grid brightness value for thecorresponding sensitivity level at the received off-grid position; anaccumulator for combining the respective sensitivity level off-gridbrightness values derived by the plurality of sensitivity level off-gridbrightness value estimator for the corresponding sensitivity levels ateach off-grid position received from the off-grid position generator toderive a respective combined sensitivity level off-grid brightness valueat each off-grid position received from the off-grid position generator;and a response function compensator for compensating the respectivecombined sensitivity level off-grid brightness value at each off-gridposition received from the off-grid position generator by the inverse ofa combined response function of the light-sensing elements to derive arespective estimated off-grid brightness value at each off-grid positionreceived from the off-grid position generator, the combined responsefunction being the sum of weighted response functions of light-sensingelements having different ones of the plurality of sensitivity levels,each one of the weighted response functions being a response function ofa light-sensing element having a respective one of the sensitivitylevels multiplied by a redetermined weighting factor for the sensitivitylevels; and an output image generator for deriving pixel brightnessvalues of an output image from the estimated off-grid brightness values.2. The system of claim 1, wherein the array of light-sensing elements ofthe image sensor is a linear array for capturing a line image and thepixel position array is a linear array having the multiplicity of pixelpositions located at respective regularly spaced positions in a pixelrow so as to form a linear captured image brightness value array, thecaptured image memory for storing the captured image brightness valuesproduced by the light-sensing elements.
 3. The system of claim 2,wherein the off-grid positions of the off-grid position array arelocated between respective adjacent pairs of pixel positions of thelinear pixel position array.
 4. The system of claim 3, wherein thepredetermined sensitivity pattern has a predetermined first sensitivitylevel and a predetermined second sensitivity level, and adjacent pixelpositions of the linear pixel position array correspond to differentones of the predetermined first sensitivity level and the predeterminedsecond sensitivity level.
 5. The system of claim 1, wherein the array oflight-sensing elements of the image sensor is a linear array and thepixel position array is a two-dimensional array having pixel positionsat respective intersections of a plurality of regularly spaced pixelrows and a plurality of regularly spaced pixel columns, the pixel rowsbeing orthogonal to the pixel columns, and wherein the linear array oflight-sensing elements of the image sensor is adapted to be sequentiallyexposed to successive regularly spaced linear regions of the scene inmultiple exposures, each one of the exposures producing a respectivelinear captured image brightness value array, the respective linearcaptured image brightness value arrays produced by the multipleexposures being concatenated to form a two-dimensional captured imagebrightness value array having respective captured image brightnessvalues at the pixel positions of the pixel position array, theconcatenated linear captured image brightness value arrays being storedin the captured image memory as respective captured image brightnessvalues at the pixel positions of the two-dimensional pixel positionarray.
 6. The system of claim 5, wherein the off-grid positions of theoff-grid position array are located at respective intersections of aplurality of regularly spaced intermediate columns and pixel rows, eachone of the intermediate columns being parallel to the pixel columns andextending between a respective adjacent pair of the pixel columns. 7.The system of claim 6, wherein the predetermined sensitivity pattern hasa first predetermined sensitivity level and a second predeterminedsensitivity level, and wherein adjacent pixel positions in each one ofthe pixel rows correspond to different ones of the first and secondpredetermined sensitivity levels, and adjacent pixel positions in eachone of the pixel columns correspond to the same one of the first andsecond predetermined sensitivity levels.
 8. The system of claim 1,wherein the image sensor has a two-dimensional array of light-sensingelements and the multiplicity of pixel positions are located atrespective intersections of a plurality of regularly spaced pixel rowsand a plurality of regularly spaced pixel columns, the pixel rows beingorthogonal to the pixel columns, the captured image memory for storingthe captured image brightness values produced by the light-sensingelements at corresponding ones of the multiplicity of pixel positions ofthe pixel position array.
 9. The system of claim 8 wherein the off-gridpositions of the off-grid position array are located at respectiveintersections of a plurality of regularly spaced intermediate rows and aplurality of regularly spaced intermediate columns, each intermediaterow being parallel to the pixel rows and extending between a respectiveadjacent pair of pixel rows and each intermediate column being parallelto the pixel columns and extending between a respective adjacent pair ofpixel columns.
 10. The system of claim 9, wherein the predeterminedsensitivity pattern of the array of light-sensing elements has fourdifferent predetermined sensitivity levels and corresponds to arepetitive pattern of groups of four nearest neighbor pixel positions,each one of the four pixel positions in each group corresponding to adifferent one of the four predetermined sensitivity levels in a commonpredetermined positional order.
 11. The system of claim 8, wherein theoff-grid positions of the off-grid position array are located atrespective intersections of a plurality of regularly spaced intermediaterows and the plurality of pixel columns, and at respective intersectionsof a plurality of regularly spaced intermediate columns and the pixelrows, each one of the intermediate rows being parallel to the pixel rowsand extending between a respective adjacent pair of the pixel rows, eachone of the intermediate columns being parallel to the pixel columns andextending between a respective adjacent pair of pixel columns.
 12. Thesystem of claim 11, wherein the predetermined sensitivity pattern of thearray of light-sensing elements has first and second predeterminedsensitivity levels, and wherein adjacent pixel positions in each one ofthe pixel rows and adjacent pixel positions in each one of the pixelcolumns correspond to different ones of the first and secondpredetermined sensitivity levels.
 13. The system of claim 1, wherein theoutput image generator comprises an output image memory for storing therespective estimated off-grid brightness values derived by the off-gridestimator as the pixel brightness values of the output image.
 14. Thesystem of claim 1, wherein the response function compensator comprises alookup table memory storing lookup table data representing the inverseof the combined response function of the light-sensing elements, and amapper for deriving from the respective combined sensitivity leveloff-grid brightness value at each off-grid position received from theoff-grid position generator a respective index for the lookup tablememory and providing a data value in the lookup table memorycorresponding to the respective index as the respective estimatedoff-grid brightness value at the off-grid position received from theoff-grid position generator.
 15. The system of claim 1, wherein theoutput image generator comprises: an off-grid brightness value memoryfor storing the respective estimated off-grid brightness value derivedby the response function compensator at each off-grid position receivedfrom the off-grid position generator; an on-grid resampler for derivingfrom the estimated off-grid brightness values in the off-grid brightnessvalue memory respective resampled on-grid brightness values at the pixelpositions of the pixel position array; and an output image memory forstoring the respective resampled on-grid brightness values derived bythe on-grid resampler as the pixel brightness values of the outputimage, and wherein the on-grid resampler comprises: a pixel positiongenerator for providing pixel positions of the pixel position array; andan interpolator for receiving pixel positions provided by the pixelposition generator and being responsive to a pixel position receivedfrom the pixel position generator for deriving from the estimatedoff-grid brightness values in the off-grid brightness value memory arespective resampled on-grid brightness value at the received pixelposition.
 16. The system of claim 1, wherein the accumulator of theoff-grid estimator comprises an adder for adding the respectivesensitivity level off-grid brightness values derived by the plurality ofsensitivity level off-grid brightness value estimators for thecorresponding sensitivity levels at each off-grid position received fromthe off-grid position generator to provide the combined sensitivitylevel off-grid brightness values at each off-grid position received fromthe off-grid position generator.
 17. The system of claim 1, wherein theaccumulator of the off-grid estimator comprises a weighting factormemory for storing a plurality of predetermined weighting factors eachcorresponding to a different one of the plurality of sensitivity levelsof the light-sensing elements, a plurality of multipliers each formultiplying the sensitivity level off-grid brightness valuecorresponding to a respective one of the sensitivity levels at areceived off-grid position by the weighting factor corresponding to therespective one of the sensitivity levels to provide a weightedsensitivity level off-grid brightness value for the correspondingsensitivity level at the off-grid position received from the off-gridposition generator, and an adder for summing the weighted sensitivitylevel off-grid brightness values at the received off-grid positionprovided by the plurality of multipliers to derive the combinedsensitivity level off-grid brightness value at the off-grid positionsreceived from the off-grid position generator.
 18. The system of claim17, wherein the array of light-sensing elements of the image sensor is alinear array and the pixel position array is a linear array having amultiplicity of pixel positions at respective regularly spaced pixelpositions in a pixel row, and wherein the off-grid positions of theoff-grid position array are each located between a respective adjacentpair of pixel positions of the pixel position array and the sensitivitylevel off-grid brightness values derived by the plurality of sensitivitylevel off-grid brightness value estimators for an off-grid positionreceived from the off-grid position generator are respective capturedimage brightness values at a predefined number of pixel positions thatare nearest neighbors to the received off-grid position, each one of thepredefined number of nearest neighbor pixel positions corresponding to adifferent one of the plurality of sensitivity levels of thelight-sensing elements.
 19. The system of claim 18, wherein each one ofthe off-grid positions of the off-grid position array is located midwaybetween a respective adjacent pair of pixel positions of the linearpixel position array and the predetermined sensitivity pattern of thelight-sensing elements has a first and a second predeterminedsensitivity level, and wherein the combined sensitivity level off-gridbrightness value at an off-grid position x′ received from the off-gridposition generator is expressed asB(x′)=W ₁ I _(c)(x′−0.5)+W ₂ I _(c)(x′+0.5), where B(x′) is the combinedsensitivity level off-grid brightness value at the received off-gridposition x′, W₁ is the weighting factor for the sensitivity levelcorresponding to pixel position (x′−0.5), W₂ is the weighting factor forthe sensitivity level corresponding to pixel position (x′+0.5),I_(c)(x′−0.5) is the captured image brightness value at pixel position(x′−0.5) and I_(c)(x′+0.5) is the captured image brightness value atpixel position (x′+0.5), and the combined response function of thelight-sensing elements is expressed asS(E)=W ₁ P ₁(E)+W ₂ P ₂(E), where P₁(E) is the radiometric responsefunction of a light-sensing element having the sensitivity levelcorresponding to pixel position (x′−0.5) and P₂(E) is the radiometricresponse function of a light-sensing element having the sensitivitylevel corresponding to pixel position (x′+0.5).
 20. The system of claim17, wherein the image sensor has a two-dimensional array oflight-sensing elements and the multiplicity of pixel positions arelocated at respective intersections of a plurality of regularly spacedpixel rows and a plurality of regularly spaced pixel columns defining atwo-dimensional pixel position array, the pixel rows being orthogonal tothe pixel columns, the multiplicity of off-grid positions are located atrespective intersections of a plurality of regularly spaced intermediaterows and a plurality of regularly spaced intermediate columns, each oneof the intermediate rows being parallel to the pixel rows and extendingbetween a respective adjacent pair of the pixel rows, each one of theintermediate columns being parallel to the pixel columns and extendingbetween a respective adjacent pair of the pixel columns.
 21. The systemof claim 20, wherein the predetermined sensitivity pattern of the arrayof light-sensing elements corresponds to a pixel position array havingrepetitive disposed groups of four nearest neighbor pixel positions,where each one of the four nearest neighbor pixel positions in eachgroup corresponds to a respective one of four predetermined sensitivitylevels in a common predetermined positional order.
 22. The system ofclaim 17, wherein the image sensor has a two-dimensional array oflight-sensing elements and the multiplicity of pixel positions arelocated at respective intersections of a plurality of regularly spacedpixel rows and a plurality of regularly spaced pixel columns defining atwo-dimensional pixel position array, the pixel rows being orthogonal tothe pixel columns, and wherein the multiplicity of off-grid positionsare located at respective intersections of a plurality of regularlyspaced intermediate rows and the pixel columns, and at respectiveintersections of a plurality of regularly spaced intermediate columnsand the pixel rows, each one of the intermediate rows being parallel tothe pixel rows and extending between a respective adjacent pair of thepixel rows, each one of the intermediate columns being parallel to thepixel columns and extending between a respective adjacent pair of thepixel columns.
 23. The system of claim 17, wherein the predeterminedsensitivity pattern of the array of light-sensing elements has a firstand a second predetermined sensitivity level, and adjacent pixelpositions in each one of the pixel rows and adjacent pixel positions ineach one of the pixel columns correspond to different ones of thepredetermined first and second sensitivity levels.
 24. The system ofclaim 17, wherein the array of light-sensing elements of the imagesensor is a linear array and the multiplicity of pixel positions arelocated at respective regularly spaced pixel positions in a pixel row soas to form a linear captured image brightness value array, and whereinthe linear array of light-sensing elements of the image sensor isadapted to be sequentially exposed to successive regularly spaced linearregions of the scene in multiple exposures, each one of the exposuresproducing a respective linear captured image brightness value array, therespective linear captured image brightness value arrays produced by themultiple exposures being concatenated to form a two-dimensional capturedimage brightness value array having respective captured image brightnessvalues at pixel positions of a pixel position array located atrespective intersections of a plurality of regularly spaced pixel rowsand a plurality of regularly spaced pixel columns, the pixel rows beingorthogonal to the pixel columns, and wherein the multiplicity ofoff-grid positions are located at respective intersections of the pixelrows and a plurality of regularly spaced intermediate columns, each oneof the intermediate columns being parallel to the pixel columns andextending between a respective adjacent pair of the pixel columns. 25.The system of claim 24, wherein the predetermined sensitivity pattern ofthe array of light-sensing elements has a first and a secondpredetermined sensitivity level, and wherein adjacent pixel positions ineach pixel row correspond to different ones of the first and the secondpredetermined sensitivity levels and adjacent pixel positions in eachpixel column correspond to the same one of the first and the secondpredetermined sensitivity levels.
 26. The system of claim 1, whereineach one of the sensitivity level off-grid brightness value estimatorsis responsive to an off-grid position received from the off-gridposition generator and the data indicative of the sensitivity levelcorresponding to each one of the pixel positions from the sensitivitypattern memory for determining a pixel position corresponding to thesame sensitivity level as the sensitivity level off-grid brightnessvalue estimator and located nearest the received off-grid position, andfor estimating a sensitivity level off-grid brightness value at thereceived off-grid position to be equal to the captured image brightnessvalue at the nearest pixel position corresponding to the samesensitivity level as the sensitivity level off-grid brightness valueestimator.
 27. The system of claim 1, wherein the image sensor is anintegrated circuit charge coupled device having an array of photodiodesin spaced orthogonal rows and columns, each one of the photodiodeshaving a light-sensing surface adapted to be exposed to incident lightand being operatively biased to store photogenerated charge therein, thecharge coupled device having respective integrated lenses formed overthe photosensitive surfaces of predetermined ones of the photodiodes,each integrated lens concentrating the incident light onto therespective light-sensing surface, whereby the sensitivity level of eachone of the photodiodes is determined by whether or not an integratedlens is formed over the light-sensing surface thereof.
 28. The system ofclaim 1, wherein the image sensor is an integrated circuit chargecoupled device having an array of photodiodes arranged in spacedorthogonal rows and columns, each one of the photodiodes having alight-sensing surface adapted to be exposed to incident light and beingoperatively biased to store photogenerated charge therein, each one ofthe photodiodes having an integrated light filter formed above thelight-sensing surface thereof, the transparency of each integrated lightfilter being fixed during fabrication of the charge coupled device,whereby the sensitivity level of each one of the photodiodes in thearray is determined by the transparency of the integrated light filterabove the light-sensing surface thereof.
 29. The system of claim 1,wherein the image sensor is an integrated circuit charge coupled devicehaving an array of photodiodes arranged in spaced orthogonal rows andcolumns, each one of the photodiodes having a light-sensing surfaceadapted to be exposed to incident light and being operatively biased tostore photogenerated charge therein, the charge coupled device beingformed with an opaque layer overlying the array of photodiodes with arespective aperture formed in the opaque layer above the light-sensingsurface of each one of the photodiodes, the respective size of theaperture above the light-sensing surface of each one of the photodiodesbeing fixed during fabrication of the charge coupled device, whereby thesensitivity level of each one of the photodiodes in the array isdetermined by the size of the aperture above the light-sensing surfacethereof.
 30. The system of claim 1, wherein the image sensor is anintegrated circuit charge coupled device having a multiplicity ofphotodiodes in an array of spaced orthogonal rows and columns, each oneof the photodiodes having a light-sensing surface adapted to be exposedto incident light and being operatively biased to store photogeneratedcharge therein, the charge coupled device having an interline structurein which the columns of photodiodes are each disposed adjacent acorresponding vertical shift register, each photodiode having anassociated transfer gate electrode for controlling the transfer ofstored photogenerated charge from the photodiode to the correspondingvertical shift register, the stored photogenerated charge in each one ofthe photodiodes being operatively periodically removed by theapplication of a periodic reset pulse to the device, the photogeneratedcharge stored in each one of the photodiodes being operativelytransferred to a respective stage of the corresponding vertical shiftregister by the application of a transfer trigger pulse to theassociated transfer gate electrode and being accumulated therein for anexposure interval, the transfer trigger pulse being applied immediatelypreceding a reset pulse, the respective transfer trigger pulses beingoperatively applied to the transfer gate electrodes associated with thephotodiodes in the array at predetermined frequencies, whereby thesensitivity level of each one of the photodiodes in the array isdetermined by the frequency of transfer trigger pulses applied to thetransfer gate electrode associated therewith.